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SUBJECT:     PSRO  Data  Quality  Control  Manual 


This  technical  assistance  document  is  intended  to  assist  PSROs  in  the 
development  and  implementation  of  a  data  quality  control  program. 

Regardless  of  whether  PSROs  have  developed  new  systems  and  procedures  to 
collect  data,  or  have  adopted  systems  in  place  prior  to  PSRO  program 
implementation  to  meet  PSRO  data  needs,  it  is  the  PSRO's  responsibility 
to  assure  the  quality  of  the  data.     The  PSRO's  credibility  and  effective- 
ness depend  upon  the  accuracy  and  reliability  of  the  data  which  are  used 
to  produce  reports  and  profiles.     These  reports  and  profiles  are  the 
basis  for  the  decisions  that  are  made  that  affect  the  management,  organization 
and  professional  activities  of  the  PSRO  and  must  be  based  on  high  quality 
data. 

As  an  aid  to  developing  a  data  quality  assurance  program,  the  manual 
provides  guidelines  and  procedures  which  PSROs  can  adopt,  modifying 
where  necessary,  and  integrate  into  their  ongoing  data  systems  and 
hospital  monitoring  programs.    The  manual  also  includes  an  excerpt  from 
a  forthcoming  publication,  "Quality  Control  of  Coding  and  Abstracting," 
to  be  published  by  the  Commission  on  Professional  and  Hospital  Activities, 
and  various  approaches  to  data  quality  control  in  the  form  of  Case 
Studies  describing  procedures  developed  by  a  number  of  PSROs. 

As  more  experience  is  gained  in  the  area  of  data  quality  control,  we 
will  refine  the  manual.    We  would  be  interested  in  receiving  comments  on 
your  experience  in  using  the  materials  and  suggestions  for  improvement. 


Michael  J.  Goran,  M.D. 
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I .  INTRODUCTION 


1.  Concept  of  Data  Quality 

With  implementation  of  the  PSRO  program  nationally,  significant 
volumes  of  PSRO  data  are  being  collected,  processed  and  used  to  monitor 
and  manage  program  activity  at  both  the  local  and  national  levels. 
Because  the  success  of  the  program  depends  to  a  great  extent  upon  the 
effective  use  of  these  data,  it  is  crucial  that  the  data  be  of  the 
highest  quality  possible. 

High  quality  data  have  the  following  characteristics: 

.  Utility  -  Data  serve  little  purpose  if  they  do  not  have  demonstrated 
utility  to  the  ultimate  users.     PSRO  data  are  useful  for  program 
monitoring  and  management  activities  at  the  local  level,  for  medical 
management  and  review  of  the  delivery  of  care  at  the  national  level, 
and  for  the  development  of  a  national  data  base  for  research.  The 
PSRO  must  have  a  thorough  understanding  of  the  information  that  is 
required  for  these  purposes  and  assure  that  the  data  are  available  in 
a  form  which  meets  the  PSRO's  requirements. 

.     Accuracy  and  Reliability  -  Profiles,  reports,  and  Medical  Care 

Evaluation  (MCE)  Study  information  generated  from  PSRO  data  will  not 
be  useful  for  either  program  or  medical  management  purposes  if  they 
are  inaccurate  or  unreliable.     Decisions  about  employee  performance, 
hospital  delegation  status,  patterns  of  medical  care,  individual 
instances  of  questionable  care,  and  numerous  other  issues  can  be  made 
only  on  the  basis  of  valid,  consistent  data  known  to  meet  established 
standards  of  quality. 

Completeness  and  Timeliness  -  Similarly,  the  completeness  and  timeli- 
ness of  PSRO  data  impact  directly  on  its  quality.     Few,  if  any, 
decisions  can  be  made  on  the  basis  of  incomplete  data.     Data  which 
are  several  months  old  may  not  reflect  the  current  operational  status 
of  the  program,  either  administratively  or  medically,  and  decision 
making  based  on  such  data  is  awkward,  at  best.     Further,  when  errors 
are  detected  in  late  data,  the  incentive  to  provide  prompt  and 
accurate  corrections  is  substantially  reduced. 

2.  Reliability  of  Hospital  Discharge  Information 

The  quality  of  data  collected  by  PSROs  has  been  and  continues  to  be  a 
major  concern  to  all  who  must  depend  upon  the  information  and  reports 
generated  by  PSRO  computer  systems.  Several  studies  have  been  con- 
ducted to  determine  the  reliability  of  hospital  discharge  data  and  to 
determine  characteristics  in  the  hospital  which  may  affect  data  quality. 
Two  studies  are  briefly  described  herein  because  the  results  of  these 
studies  have  influenced  the  concerns  and  emphasis  of  this  manual. 
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2.1    Findings  from  Re-Abstracting  Studies 


The  Uniform  Hospital  Discharge  Data  Demonstration  (UHDDD) , 
conducted  in  1972  by  the  Health  Services  Foundation  and  Bio- 
Dynamics  under  contract  to  the  Department  of  Health,  Education, 
and  Welfare,  required  the  re-abstracting  of  a  selected  sample  of 
medical  records  from  volunteer  hospitals.    Results  of  this  test, 
conducted  at  four  sites,  were  combined  with  an  analysis  of  PSRO- 
specific  data  elements  collected  during  the  PSRO  Management 
Information  System  (PMIS)  field  test  in  1976.    Exhibit  1-1  presents 
the  detailed  analysis,  by  item,  of  discrepancies  classified  by 
error  type.     The  percentage  of  abstracts  with  discrepancies  by 
item  and  test  site  are  shown  in  Exhibit  1-2. 

Exhibit  1-2  is  particularly  noteworthy;  it  indicates  clearly  that 
most  errors  occur  in  diagnostic  coding  across  the  sites.  Under- 
lying factors  associated  with  high  discrepancy  in  this  item  were: 

-  Not  applying  correct  definition  for  principal  diagnosis  or  lack 
of  familiarity  by  hospital  personnel  with  data  definitions 

-  Abstracting  from  incomplete  medical  records 

-  Diagnoses  ordering  (incorrectly  selecting  the  principal 
diagnosis) 

-  Coding  errors 

-  Incorrect  classification  of  diagnoses  (this  occurs  when  the 
terminology  used  by  the  physician  cannot  be  matched  with  termi- 
nology in  the  code  book) 

-  Two  codes  describing  virtually  identical  conditions 

Results  of  another  major  re-abstracting  study  completed  in 
February  1977,  by  the  Institute  of  Medicine,  showed 

"...that  information  on  hospital  admission  date,  discharge  date, 
patient's  date  of  birth  or  age,  sex  and  anticipated  principal 
source  of  payment  was  highly  reliable.    However,  for  all 
principal  diagnoses  combined,  when  codes  were  compared  to  four 
digits,  the  original  abstract  and  Institute  of  Medicine  re- 
abstract  agreed  for  only  65.2  percent  of  the  records;  for  all 
procedures  the  level  of  agreement  was  73.2  percent."  If 


1/  Institute  of  Medicine,  Reliability  of  Hospital  Discharge  Abstracts 

(Washington,  D.C.  :    National  Academy  of  Sciences,  February  1977),  p.  47. 


1-2 


EXHIBIT  1-1 


ITEM  DISCREPANCY  CLASSIFICATION  ANALYSIS 
(UHDDD  TEST  SITE) 
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EXHIBIT  1-2 

PERCENTAGE  OF  ABSTRACTS  WITH  ITEM  DISCREPANCIES 

TEST  SITE 
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15 

Exhibit  1-3,  excerpted  from  the  IOM  study,  shows  reasons  for 
discrepancies  between  the  original  abstract  and  IOM  re-abstract, 
when  the  IOM  re-abstract  was  determined  to  be  correct  and  when  it 
could  not  be  determined  which  data  source  was  correct. 

The  report  listed  several  factors  for  the  decreased  reliability  of 
these  data  items.     Among  these  were: 

-  Some  diagnoses  are  more  difficult  to  code  accurately  than  others; 
for  example,  chronic  ischemic  heart  disease  was  only  30.2  percent 
accurate  while  hypertrophy  of  the  tonsils  and  adenoids  97.0 
percent  accurate 

-  Erroneous  selection  of  principal  diagnosis 

-  Thoroughness  with  which  the  medical  record  was  reviewed  prior 
to  assigning  code  numbers. 

Thus  the  IOM  study  and  the  UHDDD  results  seem  to  be  in  basic 
agreement  on  the  problems  of  accuracy  with  respect  to  diagnoses. 

Other  research  in  this  area  has  been  conducted.     A  study  conducted 
by  Henrickson  and  Myers  noted  a  "significant  error  rate"  in 
recording  data  and  recommended  improvements  in  hospital  practices 
and  training.     Studies  conducted  by  the  Cooperative  Health  Information 
Center  in  Vermont  and  the  Commission  on  Professional  and  Hospital 
Activities  have  noted  the  importance  of  training  and  careful 
screening  of  completed  abstracts. 

2 . 2    Recommendations  from  Re-Abstracting  Studies 

Following  is  a  brief  listing  of  IOM  recommendations  which  are 
pertinent  to  PSRO  data  systems  and  which  significantly  influenced 
the  emphasis  of  this  data  quality  control  manual. 

-  Hospital  discharge  data  should  be  used  cautiously;  i.e.  ,  the 
degree  of  accuracy  required  depends  upon  use  of  data. 

-  "Periodic  assessments  of  the  reliability  of  PSRO  data  should  be 
included  in  the  information  reporting  requirements  of  the  PSRO 
Management  Information  System."  27 

-  "Physicians  should  be  encouraged  to  use  more  care  in  completing 
the  medical  record,  to  clearly  indicate  the  diagnosis  responsible 
for  hospital  admission...."  3/ 


2_/  Institute  of  Medicine,  p.  51. 
3/  Ibid 
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EXHIBIT  1-3 


Reasons  for  Discrepancy  by  Principal  Diagnosis  and  Correct  Source  ** 
of  Data,  Based  on  Four-Digit  Comparisons  (weighted  percent)  


Correct  data  source 
IOM  re-abstract  Indeterminate 
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diagnosis 
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renal  disease 
♦Cholelithiasis/ 

Cholecystitis 
Hernia  without 
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*The  total  number  of  cases  where  the  re-abstract  is  correct  is  slightly  less 
than  100  percent  because  for  a  few  abstracts  the  field  team  neglected  to 
determine  a  reason  for  discrepancy. 
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-  Total  review  of  the  medical  record  should  be  done  prior  to 
abstracting  information  on  diagnoses  and  procedures. 

-  Extensive  training  programs  for  hospital  personnel,  plus 
estimates  of  error  rates  in  hospital  data  should  be  provided  to 
those  collecting  PSRO  data. 

3 .  Purpose  of  Manual 

Whether  PSROs  have  developed  new  systems  and  procedures  to  collect  and 
process  data  or  have  adapted  systems  in  place  prior  to  PSRO  program 
implementation  to  meet  PSRO  review  and  data  reporting  requirements, 
it  is  the  PSRO's  responsibility  to  assure  the  quality  of  the  data. 
In  accordance  with  the  Provisional  Policies  for  PSRO  Data  Routing  and 
Processing,  "the  PSRO,  data  processing  organization,  and  delegated  hospi- 
tals must  agree  to  the  establishment  of  in-house  procedures  for  controlling 
the  quality  of  data  collected."  (3.1.4) 

In  light  of  the  deficiencies  in  Uniform  Hospital  Discharge  Data  Set  (UHDDS) 
collection  found  in  the  re-abstracting  studies  and  PSRO  responsibility  to 
establish  procedures  to  assure  and  maintain  data  accuracy  and  reliability 
in  the  collection  and  processing  of  high  quality  PSRO  data  (the  UHDDS 
plus  the  P-Specif ic  and  optional  items) ,  it  is  the  purpose  of  this  manual 
to  assist  PSROs  by  providing  guidelines  for  developing  a  data  quality 
control  program. 

The  manual  includes: 

A  data  quality  checklist  and  suggested  procedures  from  which  PSROs 
can  develop  ongoing  operational  controls  and  audit  procedures  for 
their  individual  PSRO  environments. 

Various  approaches  for  monitoring  the  quality  of  data  collected  and 
processed  through  the  PSRO's  data  system,  and  for  training  and  re- 
training personnel  in  a  positive,  helpful  manner.      These  are  attached 
in  the  appendices  in  the  form  of  Case  Studies  describing  the  data 
quality  control  procedures  developed  by  several  PSROs.     Please  note 
these  are  illustrations  of  data  quality  control  systems,  but  not  every 
aspect  of  the  systems  are  appropriate  or  even  advisable  for  another 
PSRO.     In"  reviewing  the  case  studies,  it  is  hoped  that  each  PSRO  will 
be  able  to  adopt,   (modifying  where  necessary),  those  procedures  and 
forms  which  are  appropriate  to  its  system  and  environment. 

An  excerpt  from  a  forthcoming  publication,  "Quality  Control  of  Coding 
and  Abstracting"  to  be  published  by  the  Commission  on  Professional 
and  Hospital  Activities  (CPHA) ,  is  included  as  a  method  of  data 
quality  monitoring  (Appendix  A) . 

4.  Functional  Areas  of  a  PSRO  Data  System 

Although  it  is  possible  to  categorize  the  activities  comprising  a  generic 
or  typical  PSRO  data  system  in  many  ways,  for  purposes  of  this  manual 
four  major  functional  categories  have  been  identified,  as  follows: 
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Data  abstracting  and  recording 


Forms  flow  control  and  data  transfer 
Data  entry  and  edit 

Data  processing  and  output  (report)  analysis 

These  four  functional  areas  consist  of  relatively  discrete  activities 
within  the  data  collection  and  processing  system  established  for  PSRO- 
required  data  on  hospital  discharges,  as  illustrated  in  Exhibit  1-4  on 
the  following  page.     In  this  flow  chart,  three  functional  areas  are 
depicted  as  separate  activities  while  the  fourth  activity,  Forms  Flow 
Control  and  Data  Transfer,  applies  to  each  aspect  of  the  other  three 
major  activities.     The  number  in  parentheses  under  each  activity 
references  a  functional  area  and  activity  with  the  related  quality 
control  procedures  in  the  PSRO  Data  Quality  Control  Checklist  in 
Chapter  II.     Each  of  the  four  functional  areas  is  described  in  more 
detail  below. 

Data  Abstracting  and  Recording 

The  data  abstracting  and  recording  function  generally  occurs  within 
the  hospital.     It  usually  involves  the  review  coordinator  and  the 
medical  records  department,  and  in  some  cases  also  involves  personnel 
at  the  PSRO.     Specific  activities  which  usually  occur  within  this 
function  include: 

.  Identification  of  patients  to  be  reviewed 

.  Documentation  of  concurrent  review  information,  including  extension 
denials,  physician  reviews,  number  of  days  certified,  reason 
codes,  etc. 

.  Documentation  of  relevant  medical  information,  dates,  and  other 
essential  identifying  information  on  the  review  coordinator's  work- 
sheet and/or  in  the  medical  record 

.  Abstraction  of  the  PHDDS  or  PSRO-required  information  onto  the 

abstract  form  required  by  the  PSRO  to  be  completed  for  each  Title  V, 
XVIII,  or  XIX  patient 

.  Identification  of  the  principal  and  other  diagnoses  and  principal 
and  other  significant  procedures 

.  Coding  the  procedure (s)  and  diagnoses  in  H-ICDA-2  or  ICDA-8  on  the 
abstract  (this  may  also  occur  at  the  PSRO) 

.  Certification  of  the  patient's  bill. 

In  short,  this  function  involves  identifying  the  patient,  documenting 
the  case,  and  abstracting  and  recording  the  data  for  PSRO  purposes. 
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EXHIBIT  1-4 
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EXHIBIT  1-4 


PSHO  DATA  SYSTEM  INDICATING  ON-GOING  DATA  QUALITY  CONTROL  PROCEDURES 


I 


I 


FLOW  CHART  NOTE:     Two  options  have  been  presented  on  the  flow  chart. 
In  Option  1  the  hospital  sends  abstracts  to  the  PSRO.     In  Option  2 
abstracts  are  sent  to  the  abstract  service  organization.     There  are 
other  options,  such  as  the  combination  of  Options  1  and  2,  which  are 
not  depicted  here.     Individual  activities  may  vary  as  follows: 

.  The  admission's  office  may  use  another  method  to  notify  the  review 
coordinator  other  than  patient  listings 

.  The  PSRO  Review  Coordinator  function  may  be  delegated  or  non-delegated 

.  The  review  coordinator  may  not  perform  any  screening  activity. 
In  some  large  hospitals,  only  a  review  coordinator  supervisor  may 
perform  this  activity. 

Forms  Flow  Control  and  Data  Transfer 

Forms  flow  and  data  transfer  activities  occur  whenever  information, 
which  has  been  recorded,  processed,  or  otherwise  handled  by  one 
individual  or  entity,  is  passed  to  another.     Forms  flow  occurs  when  the 
data  is  recorded  on  a  form,  such  as  an  abstract  or  a  review  coordinator's 
worksheet.     Data  transfer  refers  to  the  movement  of  data  other  than  in 
hard  copy  form,  such  as  transmittal  of  the  data  to  the  processor  from 
the  PSRO  via  cathode  ray  tube  (CRT) .     Control  of  each  of  these 
activities  is  an  integral  part  of  the  data  system  and  involves: 

.  Controlling  all  documentation  and  input  forms  required  for  or  gener- 
ated as  a  result  of  the  hospital  stay,  such  as  concurrent  review  or 
billing  certification. 

.  Controlling  all  data  transfer  to  and  from  the:    hospital,  collector, 
processor,  and  PSRO. 

Data  control  must  be  maintained  regardless  of  data  format  which  may 
include : 

.  Hard  copy  input  (abstracts)  and  output  (reports  or  listings) 
.  Magnetic  tape  input  and  output 

.  Hard  copy  and  tape  combinations  for  input  and  output 

.  Transmissi  on  of  data  from  one  place  to  another  via  CRT  (telephone 
lines) 

.  Other  data  entry  media,  such  as  diskettes,  punched  cards,  etc. 

The  forms  flow  control  and  data  transfer  function  are  important  for 
quality  control,  because  errors  or  problems  can  be  introduced  into  data 
at  this  stage;  abstracts,  records  or  data  items  may  be  lost  or  never 
reported  at  all;  data  contents  may  be  transposed  or  changed  inadvertently 
during  transmission;  or  incomplete  processing  may  allow  the  entry  of 
errors  into  the  PHDDS  data  base. 
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FLOW  CHART  NOTE:     The  functional  category  relating  to  forms  flow 
control  and  data  transfer  is  found  within  each  major  activity  of  the 
flow  chart  (Exhibit  1-4).     Procedures  for  control  of  data  transfer  are 
basically  the  same  wherever  the  data  are  located;  i.e.,  logging,  batching, 
verifying  the  sending  and  receiving  of  information. 

Data  Entry  and  Editing 

Data  entry  and  editing  functions  involve  the  process  of  keying  or  other- 
wise entering  (such  as  use  of  an  optical  scanner)  the  PHDDS  data  into 
the  computer  for  processing;  the  function  may  be  performed  at  the 
hospital,  at  the  PSRO,  by  the  collector,  or  by  the  processor.  Editing 
includes  the  systematic  checking  of  the  entered  PHDDS  records  according 
to  a  predetermined  series  of  standards  by  the  computer  either  during 
or  after  data  entry. 

The  editing  portion  of  this  function  may  require  that  the  forms  flow 
control  and  data  transfer  function  again  become  relevant  if  hard  copy 
error  lists  require  the  preparation  and  transfer  of  corrected  abstracts. 
If  an  interactive  CRT  is  the  data  entry  method  used,  much  editing  occurs 
on-line  while  the  data  are  being  entered,  thereby  decreasing  greatly 
the  necessity  for  hard  copy  correction  abstracts. 

FLOW  CHART  NOTE:     Data  entry  activities  are  performed  by  one  or  more 
of  the  following  participants:     the  PSRO  data  entry  personnel,  an 
abstract  service  organization  or  the  PSRO  data  processor.  Editing 
performed  by  the  PSRO  or  abstract  service  organization  becomes  a  part 
of  data  entry  and  editing  as  opposed  to  final  editing  performed  by  the 
processor.     Once  again,  two  portions  are  shown  for  data  entry: 
(1)  on-site,  or  (2)  by  the  data  processor.     Pre-screening  can  also  be 
performed  by  any  of  these  participants.    All  participants  are  involved 
in  data  control  and  transfer. 

Computer  Data  Processing  and  Output  Analysis 

The  computer  data  processing  function  occurs  following  entry,  editing 
and  correcting  of  each  record.     It  is  almost  exclusively  carried  out 
by  the  processor  and  involves: 

Development  of  files  or  a  data  base  for  each  PSRO  containing  PHDDS 
and  optional  items  collected  by  the  PSRO 

Routine  updating  of  the  data  base  as  more  information  is  received 
and  corrected  (usually  on  a  monthly  basis) 
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.  Production  of  hard  copy  reports  for  the  PSRO  in  accordance  with  the 
terms  of  the  contract  between  the  PSRO  and  the  processor  (and  in 
some  instances,  preparation  of  the  PSRO  Review  Activity  Summary  for 
the  Health  Standards  and  Quality  Bureau  (HSQB) 

.  Preparation  of  the  PHDDS  tape  for  HSQB  on  a  quarterly  basis 

Also  included  within  this  functional  area  are  the  review  and  analysis 
of  the  PSRO's  reports  and  the  determination  of  the  types  and  causes 
of  error  which  are  in  the  data  (identified  as  a  result  of  studying  the 
reports) . 

This  last  function  provides  the  basis  for  the  quality  control  mechanisms 
which  are  to  be  applied  during  the  four  functional  phases  of  the  PSRO 
data  system.     The  analysis  of  problems  —  techniques  which  are  summarized 
in  the  following  section  of  this  chapter  —  is  followed  by  development 
of  action  plans  for  correcting  and  monitoring  the  process  to  assure 
data  quality. 

The  review  coordinator  supervisor  is  a  PSRO  employee  who  oversees  the 
concurrent  review  process  in  all  PSRO  hospitals.     This  person  would 
probably  be  responsible  for  implementation  of  ongoing  data  quality 
controls  based  upon  reports  from  the  data  analyst. 

5.    Application  of  Data  Quality  Control  Procedures 

In  controlling  data  quality,  two  types  of  procedures  are  needed  to 
assure  that  problems  are  systematically  identified  and  addressed: 

.  Operational  controls,  which  become  part  of  ongoing,  daily  operations 

.  Periodic  audit  controls  to  identify  data  problems. 

These  types  of  control  procedures  differ  in  the  frequency  with  which 
they  are  performed  and  in  the  techniques  used.     The  two  are  briefly 
described  below. 

Operational  Controls 

Operational  controls  are  ongoing  procedures  designed  to  assure  and 
monitor  data  quality  as  a  part  of  the  daily  functional  activities 
comprising  the  PSRO  data  system. 

Responsibility  for  performing  operational  controls  is  shared  among  the 
many  individuals  performing  the  various  PSRO  tasks  and  activities. 
Included  among  these  persons  are: 
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.  PSRO  staff  personnel,  including  coders,  data  entry  personnel, 
reports  analysts,  etc. 

.  Review  coordinators  in  delegated  and  non-delegated  hospitals 

•  Hospital  medical  records,  admitting  office,  and  billing  office 
personnel 

.  Collector  personnel,   (i.e.,  an  abstract  service  organization) 

including  mail  clerks,  data  entry  personnel,  computer  personnel,  etc. 

.  Processor  personnel  in  roles  similar  to  those  for  the  collector. 

When  operational  controls  are  needed,  and  the  exact  procedures  for 
their  conduct,  including  frequency  as  well  as  technique,  are  defined, 
the  point  in  the  data  system  at  which  they  should  be  conducted  and  the 
individual  to  perform  them  must  be  determined.     Among  the  considerations 
in  deciding  upon  these  are  the  attitudes,  capabilities,  and  current 
operating  procedures  of  the  relevent  organizations  involved,  and  the 
design  of  the  PSRO's  data  system. 

For  example,  screening  hard  copy  abstracts  for  omission  in  required 
data  fields  prior  to  data  entry  may  be  performed  at  a  number  of  points 
in  the  PSRO  data  system: 

.  Within  the  hospital,  prior  to  sending  the  abstracts  to  the  collector 
or  to  the  PSRO 

.  By  the  data  collector  prior  to  data  entry 

.  By  the  PSRO  prior  to  sending  the  abstracts  to  the  collector  or 
processor 

.  By  the  PSRO  prior  to  data  entry 

.  By  the  processor  prior  to  data  entry  by  a  combination  of  the  above. 
Audit  Procedures 

Audit  procedures  are  periodic  activities  and  tests  conducted  by  the 
PSRO  to  review  the  effectiveness  of  the  operational  controls  currently 
being  applied  and  to  identify  additional  controls  needed  to  assure  data 
quality.     In  some  cases,  audit  procedures  may  be  successfully  combined 
with  hospital  assessment  and  monitoring  activities  which  occur 
periodically.     This  combination  can  be  particularly  effective  for 
re-abstracting  medical  records. 
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Audit  procedures  are  the  responsibility  of  the  PSRO.     PSRO  personnel 
will  determine  the  type  and  frequency  of  the  audits  needed  and  may, 
in  some  instances,  retain  consultants  or  experts  for  assistance  in 
particular  technical  areas  (for  example,  in  recording  diagnoses  and 
procedures) . 

The  frequency  with  which  audits  are  performed  will  depend  upon  several 
factors,  such  as  the  amount  of  time  required  to  perform  the  audit,  the 
past  performance  of  hospitals,  and  the  results  of  the  analysis  of 
output  reports  prepared  by  the  processor  for  the  PSRO. 

Key  Considerations  of  a  Data  Quality  Control  Program 

In  designing  and  implementing  a  data  quality  control  program,  certain 
key  factors  which  must  be  considered  by  the  PSRO  are: 

.  The  program  should  continuously  monitor  data  quality 

.  The  program  should  emphasize  identification  of  patterns  of  errors 
rather  than  single,  random  errors  occurring  sporadically 

.  The  program  developed  should  actively  involve  the  hospitals  in  the 
data  quality  control  process 

.  Written  procedures  should  be  developed  which  are  understandable, 
straightforward,  and  flexible  in  operation.     These  should  be 
distributed  to  all  PSRO  and  hospital  personnel  involved  in  data 
activities 

.  The  program  should  be  oriented  toward  the  PHDDS  and  locally  defined 
optional  items  related  to  the  concurrent  review  process  and  should 
provide  clear  definitions  for  all  of  these  data  elements 

.  It  should  be  possible  to  implement  the  program  with  a  minimum  of 
technical  expertise  in  data  processing  and  coding 

.  It  should  be  possible  to  operate  the  program  with  available  staff 
resources  and  time.     Data  quality  control  functions  should  be 
integrated  with. other  PSRO  monitoring  activities 

In  addition,  the  following  factors  are  necessary  for  any  credible 
quality  control  program: 

.  The  PSRO  must  have  a  thorough  understanding  of  data  procedures  and 
activities  for  each  of  the  hospitals  in  its  area  regardless  of  the 
delegation  status  of  those  hospitals,  and 

.  The  PSRO  must  have  personnel  who  have  been  trained  in  each  quality 
control  function 
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.  The  PSRO  should  develop  minimum  standards  for  supporting  documentation, 
data  accuracy,  and  data  timeliness  for  both  the  review  process  (work- 
sheets, abstracts)  and  for  the  flow  of  PSRO  data  (logs,  batch  control 
documentation).     The  hospitals  should  be  apprised  of  the  standards 
against  which  their  performance  will  be  judged.    Review  of  hospital 
performance  in  these  areas  should  be  integrated  with  the  PSRO's 
other  monitoring  activities  if  possible. 

PSROs  should  actively  assist  delegated  hospitals  in  the  development 
of  internal  data  quality  control  procedures,  which  the  delegated  hospital 
is  required  to  establish  (PSRO  Data  Routing  and  Processing  Policy 
Number  3.1.4).    PSROs  may  also  wish  to  consider  the  presence  of  an 
effective  data  quality  control  system  as  a  delegation  criterion. 

The  PSRO  should  establish  minimum  standards  for  education  and/or 
training  for  all  persons  who  will  be  abstracting  and/or  coding  PSRO 
data.    The  PSRO  may  also  wish  to  establish  this  as  a  delegation  criterion. 
Such  training  should  be  made  available  often  by  the  PSRO  from  qualified 
instructors. 
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CHAPTER  II 


ONGOING  OPERATIONAL  DATA 
QUALITY  CONTROL  PROCEDURES 


II.     ONGOING  OPERATIONAL  DATA  QUALITY  CONTROL  PROCEDURES 


This  chapter  discusses  ongoing  operational  data  quality  control  procedur 
and  provides,  in  the  form  of  a  chart,  a  checklist  for  PSRO  data  quality 
control  from  which  the  PSRO  should  be  able  to  verify  the  integrity  of 
its  data  system,  identify  potential  sources  of  error  and  problem  areas 
and  formulate  ongoing  data  quality  control  procedures  most  appropriate 
to  its  operational  data  system  and  environment. 

Data  Quality  Control  Checklist 

The  following  data  quality  control  checklist  is  structured  in  terms  of 
the  four  functional  areas  described  previously: 

1.  Data  abstracting  and  recording 

2.  Forms  flow  control  and  data  transfer 

3.  Data  entry  and  editing 

4.  Data  processing  and  output  (report)  analysis 

These  areas  are  further  divided  into  sub- functions.     The  PSRO  should 
idenfity  those  functional  components  appropriate  to  its  particular  data 
system  and  use  the  related  material  to  design  and  implement  a  comprehen- 
sive data  quality  control  program  or  refine  an  existing  program. 
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EXHIBIT  1 1- 2 


SAMPLE  SCREENING  GUIDELINES 


DATA 
ELEMENT 


CRITERIA 


NOTES 


Hospital  No 


Field  must  be  complete 
Valid  No. 

Each  hospital  has  unique 
No. 

A  batch  of  input  forms 
from  one  hospital  should 
contain  only  one  hospital 
number. 


Hospital  can  verify. 
PSRO  can  check  this  number  when 
abstracts  are  received. 
Computer  can  audit  for  valid  hos- 
pital number  for  each  batch. 


Record  No. 


Valid  No. 

(Should  screen  for  format 
and  possibly  logical 
number  assignment) 


Hospital  screen  for  format  of 
number.     This  field  has  a  high 
accuracy  level  because  it  is 
a  highly  used  data  element 
in  the  hospital . 


Physician 
Numbers 


Review 

Coordinator 

Number 


Patient 
Identifica- 
tion 


Valid  No. 

Screen  for  correct  format. 
License  Nos.  if  used  should 
fall  within  set  numbers, 
i.e.,  Low  numbers  would  be 
deceased  physicians.  High 
nos.  not  yet  assigned. 


Must  be  valid. 

Hospital  may  have  one  or  more 
numbers. 

Unique  numbers  for  each 
hospital . 


Must  be  valid 
code . 

Social  Security  Nos.  have 
only  9  numbers  and  no  more. 
Medicaid  and  Medicare  Nos. 
have  a  set  format  which  can 
be  screened  quickly. 


Hospital  screen  particularly 
assigns  unique  numbers.  If 
license  number  is  used,  hos- 
pital, PSRO,  and  processor 
can  verify  that  numbers  fall 
within  certain  parameters  and 
take  a  certain  format  (i.e. , 
all  numerical)  Check  with 
State  Licensing  Board  for 
parameters  for  numbers. 


Hospital  can  screen  quickly 
for  this. 

PSRO  would  take  more  time  to 
screen.     Computer  can  audit: 
certain  numbers  for  each 
hospital . 


Hospital  screen  test,  parti- 
cularly if  Patient  Chart 
Number  is  used  instead  of 
Social  Security  Number. 


EXHIBIT  1 1 -3 


ERROR  FIELD  REPORT 

lospital  I.D.#:    Total  #  Of  Abstracts   

Coordinator  I.D.#:    Total  #  of  Discrepancies 

iatch  #  :    %  of  Abstracts  with  discrepancies 

iatch  Date: 


ITEM 

NO.  OF 
ERRORS 

MOST  FREQUENT  OCCURING  ERROR  TYPES 

CHECK  MOST 

FREQUENT 

ERRORS 

>ERSON  I.D. 

3ATE  OF  BIRTH 

TRY 

RACE 

ZIP  CODE 

ADMISSION  DATE 

ADMISSION  HOUR 

DISCHARGE  DATE 

OPERATING  PHYSICIAN 

PRINCIPAL  DIAGNOSIS 

OTHER  DIAGNOSES 

PRINCIPAL  PPfTTDURE 

OTHER  PPOCEDURES 

PROCEDURE  DATES 

PRINCIPAL  SOURCE 
OF  PAYI4EOT 

DISPOSITION  OF  PT. 

(n 
co 

►H 
►J 
X 

o 
w 

o 

►J 

o 


z 
o 
o 


< 

3 

o- 

< 
a 

o 

OS 


CO 
Cd 

a; 

a 
cu 
u 
o 

04 
Ou 

■J 

o 


2 

o 
o 


H 
O 
< 


iJ 
< 
Z 

o 

M 

H 
O 

z 

(14 


g 

QJ  O 

Ol  u  u 

04  rH] 

3 

01    CD  O 

JS   cd  -H 

U  <M 

CO  4h 

ON  IH  iH 
JOT) 
44 

■o  «  « 

41   d  M 

c  -h  a 

■H  73 

IS   l-l  JS 

u  o  o 

c  o  i-i 

i-i  o  js 

I  3  § 

01  01 

CO  iH  73 

00  >  i-H 

o  at  « 

iH  M  i-l 
41  CO 

act) 

O   O  CO 

JS  i-l  o 

o.  u  J3 
4i  a 

rH  41 

4i   3  41 

u  trc 

iH 

3  41  B 
41  JS  h 
•H  44  41 
>  44 
41  44  41 

04  O  73 


O 

44  fl 
JS 

00  o 

•H  73 
!-> 

co 

U  44 

o  o 
at 

u  u 

u-l  4-1 

4)  at 

H  jo 

o 

4-1  73 

H  § 


•a 

CO  rH 

rH  41 
CO  i-t 


.  a,  a) 

41    CO  4J 

u  o  <g 

41  JS  73 
l-l 

Cu  S>n  >n 

s  e  <w 
o  a  in 
o  g  u 
oo 

O  44  3 
u  H  -n 


co  d 

■H  41 
JS  41 

u  u 

<j 

41  CO 
N 

■H  J>i 
QO  M 

cd  a) 
JS  d 

H 

41  i-l 
r-( 
-  41 
CO  14 

i-i  a. 

JS 

4-1  4) 
JS 
44  4-> 

o 

(3 

41  i-l 
CO 

3  41 

CO  u 
U  C 
4)  41 
40  I-i 
U 

co  3 

4J  CJ 

■H  CJ 
73  O 
41 

U 
co  O 
CO  U 

a  u 

CU  41 


i-i 
.  o 

00  4J 

G  O 
•H  CO 

o  e  <4-i 

4-1  1-1 

CO  U 

41  M  41 

3  4J  JS 

73  4J 
4J  O 

41  d 

43  41  l-l 

i-l  O 
>>  CJ 


14 

O 

U  144 

l-l  O 

41 

tJ  CJ 

4i  ct) 

JS  i— 1 


44  CO 

44  41 

3  14 

co  3 
•a 

41 


o 
o 

M 
Cu 

a 

41 
•  4-1 
CO  4-1 
O  iH 

%  £ 


co 

•H  00  144 

a  4i  o 

14  rH 

41  41  J4 

41  M  U 

4)  cd  CO 

Q  CJ  i-H 


a 
a 

4)  • 

rH  U 

41  41 
4-1 

cd  3 

4J  a 

cd  a 

-a  o 

CJ 

JS 

O  41 

•H  JS 

41  J3 

s 

i-l  73 

e  s 

41  iH 

4->  73 

41  3 

Q  cd 


cd 

JS 

M    44  41 

O  JS 

41  4-1 

•  14 

oo  3  a 

14  m  i-i 

0  a 

V4   41  >. 

14  i-H 

01  •-  44 

oo  d 

41    8  41  • 

O    U  3  4J 

CU   O  CT  i-l 

JS  44  CU  73 

44  00  3 

3  44  cd 

14   41  3 

O  <44  00  >» 


JS 
00 
3 

o 

M 


cd 

§  d 

14  O 

o 

<44  00 
X 
44  CJ 
3  4) 

a  js 
d  o 


d  o 
4i  a, 

4)  CO 
14 

u  o 

CO  73 


13  i*. 

4)  iH 

S  3 

41  O* 

m 

73  41 

73  > 

«  iH 


CJ 
41 

a. 

00 

o 
u 

44 

4) 
M 


I 


M 

CJ 

CJ 

cd 

14 

3 

cj 
o 


o 

CO 

u 

3 

CJ 

CJ 

cd 
cd 

44 

cd 

73 
14 

o 


a  4) 
oo  js 

d  44 

•H 

<44    44  144 

o  a  o 

•H 

4J  1 

14    44  O 

O    d  CO 

a.  4i 

41  -H  41 

14    44  C 

at  i-t 

a)  E 

OHO 

•H    CO  44 

44    3  CU 

CO  73  73 
•H  iH 

44    >  d 

CO  -rl  at 

44  73  O 

00  d 

i-l  o 

at  at 

d  73  co 

■h  d  a- 

44  cd 

3  eg 
o  cd 

14    44  • 

Cd  CO 

>44  73  CU 

O  X 

41 

41   at  73 

00   44  d 

3  -H  -H 

a. 

41   CO  73 

JS    O  d 

44  JS  CO 


$3 

fH  O 

cr  cd 
<  > 

41 
JS 

•  44 
00 

01  00 

-h  d 

CJ  i-t 

d  * 

CO  <44 

a.  t-i 

41  44 

i4  d 

CJ  41 
CO  73 


a0 


J3  t-l  -H 

Cd  73 

73   44  d 

41  i-l  II 

Qi  PU  CL 

O    CD  Pa 

•H  O  <! 
41  30 

>  d 

41  73  i-l 
73  d 

3  73 

d  cu 

41  rH  44 


73 

M 

00  o 

d  <44 

1-1 

U  >y 
41  00 
73  O 
U  tH 
O  O 
73 

35 

O  44 
iH  CU 

>  a 

J2 

O  3 

41 

41  i-l 

14  > 

O  41 

a  »4 


1-4   CU   cd  cd 

44   Jj    C  CJ 

o  o 

CO  i-l  t— 1 

at  co 

JS  oo  CO 

41  i-l 
CO  44 

M   O  73 

o  14  d 

14  04  CO 
14 

«    C  ^ 

o  < 

00  X 

d  d  p4 

■HOC-} 

73  iH  ^ 
O  CO 

CJ    CO  CO 

-H  4) 


u 

rH  at 
3  73 

<44  |4 

4)  O 
CO 

3  at 

M 

CU  3 
J3  73 
>-.  41 

I  g 
M 

00  o, 

44 

14  73 

o  d 

a.  at 
at 

14  CD 

iH 
■44  00 

o  o 
d 

oo  oo 
at  <d 
an 

>»73 
44 

00  •• 
00  d  CO 

d  i-i  at 

i-l  >>  -H 

3  <44  O 
OHO 


I 


O    B  44  rH 

O  iH 

00  CJ  > 

73  -H 

d  at  44 

•H  JS  CJ 

M   44  <J 


d 
o  at 

<44  73 


at 

at 

M 

cj 
00 

d  i-i 

■H  73 


3 

o 

JS 
JS  CO 

CJ 

■H  ai 

JS  14 

3  3 
73 

44  at 
3  CJ 

o  o 

J4  14 

cd  cu 

>>  14 
44  O 

•H 

d  co 

iH  -H 

a)  co 

44  O 

u  a 

41  00 

cj  cd 
d  ih 

3  73 


CJ 

d 

iH 

u 

Cu  CO 
■H 

eg  oo 
JS  o 

44  d 

00 
oo  co 
a)  iH 

73 

73 

41  14 
44  4) 

at  js 

d  44 

00  o 
•H 

CO  14 

01  o 

73 

i-H 
41  at 
J3  Cu 


■44 

o 

CO 

c 
o 


/-»  d 

CO  iH 

IH  <4H 

o  at 

IH  73 
14 

ai  d 


at  co 
4J  d 
d  o 

41  iH 

a  44 

oo  at 

73  iH 
3  14 

i—»  at 
«-/  > 


73 

d 

cd  co 

14 

.  «  o 
co  co  Ih 
ecu 
o  ca  at 

iH  -H 
44  CJ  rH 
Cd  iH  Cd 
N  00  U 
iH  >,  3 
d  JS  73 
Cd  CU  41 
00  CJ 
14  -  O 
O  00  14 
rH  O. 

at  at  n— ' 

O  44 
iH  iH  CO 

>  a.  o 

14    CO  OS 

cu  o  co 

CO  JS  P4 


00 

d 

■H 

|4  44 

■H  73  O 
3  41 

O*  44  44 

41    CO  at 

IH   -H  41 

rH  JS 

oo  co 
at  co 

MHO 

3  CO  cj 

73   O  cd 

at  d  44 

CJ  00 
OSS 

U  iH  JS 

O.  73  44 

rH    01  d 

at  JS  o 


CU  44  CO 

00   CO  u 

O  JS  iH 

EC    44  4H 


73 

14  i-H 

O  cd 

CJ  14 

at  3 

IH  73 

at 

41  O 

J4  O 

u 

73  Cu 

14  - — 
O 

O  rH 

cu  a) 

M  Cu 
iH 

rH  CJ 

ca  d 

O  iH 

iH  IH 

73  Cwi 

at  co 

a  co  14 
co  o 

at  u 

JS  73  44 

44  ai  ai 


rH 

•H 

U 

k 

cd 

73 

00 

at  d 

o 

at 

CU 

44 

at 

CO 

d 

44  iH 

44 

JS 

GO 

44 

iH 

44 

rH 

iH 

Cd  4H 

e—  44 

•H 

CD 

Cu 

a 

CO 

at 

73  at 

at 

at 

Ss 

co 

-a 

3 

J3 

73 

a> 

14  14 

44 

CD 

o 

-H 

73 

00 

M 

rH    3  O 

CO 

JS 

M 

•H 

41 

d  at 

3 

CO  73  44 

JS 

o- 

00 

44 

> 

C/l 

14 

•H  JS 

73 

Cu  41 

3 

60 

c 

at 

44 

•H 

at 

73  44 

cu 

•H   CJ  41 

c 

-H 

JS 

at 

73 

z 

O 

CJ 

CJ    O  rH 

73 

•H 

14 

44 

C 

a 

o 

00 

CJ  73 

o 

S    H  Jl 

a 

c 

IH 

OJ 

-.H 

CU 

rH 

d 

d 

U 

■H   Cu  iH 

CO 

iH 

3 

d 

J3 

44 

H 

o 

oo  ca 

o. 

14  00 

cd 

U 

i-i 

at 

CO 

CO 

iH 

d  44 

O.T3  d 

d 

IH 

CJ 

JS 

00 

>v 

bJ 

44 

O  00 

at 

d  o 

o 

44 

o 

44 

M 

o 

CO 

3 

iH 

CO  u 

JS 

00  CO   CU  iH 

iH 

o 

a 

cr 

d 

U  41 

44 

d  oo 

44 

14 

co 

3 

|4 

at 

iH 

41  73 

-H   00  41 

cd 

01 

M 

M 

M 

JS 

4H 

o.  d 

eg 

d 

JS 

o 

r*» 

01 

O 

44 

at 

3 

M 

•H  ca 

•H 

44 

rH 

73 

rH 

c— 

at 

a  o  oo 

a 

44 

44 

rH 

d 

01 

c 

rH  >> 

CO 

u  a  in 

14 

o 

01 

CO 

CO 

IS 

iH 

44 

73 

CO  rH 

C 

44 

4)  00 

cu 

CJ 

o 

O 

JS 

cd 

4! 

rH 

O 

CO 

44    CO  O 

44 

CO 

cu 

■H 

44 

00 

O  3 

■H 

41  iH  JS 

01 

iH 

!h 

44 

M 

o 

•H 

£ 

3 

O  44 

44 

73  73  5 

73 

JS 

■< 

CO 

O 

44 

* 

o 
vi 

C/l 
Cu 

oo  - 

c 

-H 
73 
U 

O 

u 

OJ 


73 
41 
3 

d 

a  ih 
d 


c 

o 


01 

14 

3 
73 

OJ 

CJ 

O 
IH 


CO 
W 
OS 
» 
O 
W 
CJ 

o 

OS 
CU 

o 

e 

z 
o 
o 


< 
o- 


1 

i 

X) 

IB 

0! 

CO  .C 

4J 

«J 

o 

C 

0) 

to 

Xi 

cu  4J 

• 

CU 

<u 

td 

1 

i 

CO 

a 

CO 

X 

o 

•H  iH 

4J 

M 

Xi 

VI 

XI 

d 

rH 

o 

■H 

Oil 

4-> 

e 

TJ 

>. 

Vi  S 

o 

a 

o 

a) 

4-1 

4-1 

rH 

c 

rH 

03 

CO 

4J 

00 

a 

3 

cd 

e> 

MH 

a 

a 

•H 

00 

iH 

0) 

00  1-1 

CO 

to 

a 

1 

4J 

a 

CO 

CO 

XI 

tv 

01 

id 

n 

00 

CO 

C 

> 

•H 

S  VI 

td 

tu 

-a 

tj 

u 

CO 

CD 

B 

•H 

a 

u 

vi 

iH 

•H 

TJ 

a) 

CO 

3  O 

3 

M 

4) 

u 

c 

O 

CD 

td 

3 

CU 

CO 

•a 

oo 

4J 

CIS 

o 

3 

4-1 

(U 

00  » 

4J 

a. 

<o 

o 

ai 

XI 

o 

cn 

4J 

1 

M 

XI 

X 

U 

o 

o 

Vl 

td 

C 

•H 

td 

o 

t4 

a. 

o 

tU 

4J 

•o 

o 

CD 

o 

■a 

rH 

O 

CJ 

U 

CO 

x 

CO 

O 

■a 

-H 

cu 

CU  T3 

to 

rH 

pd 

01 

rH 

Vl 

d 

•a 

XI 

cd 

d 

rH 

a. 

CO 

CO 

CD 

10 

CD 

MH 

rH 

Vl 

00  iH 

u 

CU 

H 

Vl 

00 

a. 

XH 

o 

td 

to 

4J 

Vl 

td 

cu 

CD 

■H 

C1J 

oo 

00 

X> 

CU 

M  3 

CO 

o 

> 

d 

s 

O 

>4H 

4J 

00 

vi 

00 

-o 

00 

o 

a 

td 

c 

CO 

XI 

td  O 

i-l 

tu 

cu 

4-1 

CO 

4-1 

d 

VI 

CO 

Vi 

c 

o 

CD 

iH 

•H 

1-1 

CU 

a 

X!  XI 

X! 

• 

3 

VI 

to 

CU 

TJ 

•rl 

-H 

o 

XI 

d 

(X 

CD 

1 — 1 

1-1 

4_) 

4J 

Vl 

4-1 

TJ 

CJ  CO 

4J 

td 

a. 

at 

> 

tU 

to 

4-1 

MH 

cd 

iH 

3 

X 

.-1 

1 

4-1 

P. 

CJ 

CU 

4-1 

to 

to 

4J 

cd 

U 

4-1 

-a 

td 

•H 

4J 

■H 

VI 

rfj  • 

0 

O 

at 

to 

o 

CO 

4-1 

iH 

cu 

■H  O 

CU 

id 

•H 

cd 

3 

u 

4J 

CJ 

> 

o 

to 

ri 

O  ^"N 

Li 

to 

4-J 

•H 

4J 

Go 

Vl 

Vi 

•H 

a 

to 

CO 

13  OS 

> 

T3 

VI 

Vl 

o 

o 

* 

CO 

CU 

a. 

tu 

1 

CJ  00 

00  iH 

00  iH 

< — t 

iH 

CU 

4J 

o 

Vl 

to 

J-J 

CO 

o 

CU 

3 

4-1 

rH 

00 

D.  rH 

m 

d 

•a 

a 

X> 

Vl 

CD 

4J 

O 

3 

pC 

iH 

0)  Ot 

VI 

rH 

4-> 

CJ 

MH 

d 

cu 

d 

tU 

C 

3 

MH  iH 

■H 

TJ 

•* 

CU 

<d 

to 

H 

a.  4J 

a 

td 

•H 

CJ 

•H 

iH 

cu 

to 

•5 

to 

MH 

MH 

O  CJ 

u 

i — i 

a. 

4-1 

CSJ 

to 

CO 

CO 

13 

a 

CD 

cu  cu 

B 

4-1 

Vi 

id 

> 

CD 

8 

O 

d 

3 

i— i 

3 

CD 

•H 

4-1 

a) 

3 

iH 

CU 

Vl 

O 

r-i  x; 

•H 

■H 

a 

CU 

to 

CU 

CU 

0) 

o 

-3 

CU 

to  tu 

-a 

CO 

O 

■h 

<4h 

U 

x 

i_i 

XI 

TJ 

4-1 

O 

• 

-C 

a.  H 

a. 

00 

CJ 

CU 

Vl 

Vl 

XI 

o 

§ 

X! 

d 

C  3 

ai 

1-1 

rH 

■H 

o 

u 

CO 

c 

Vl 

4-1 

4-1 

CO 

a 

O 

to 

0) 

d 

■H 

-H 

o. 

a. 

4J 

o 

vi  cr 

o 

00 

00 

O 

a. 

4J 

CO 

O 

•H 

u 

c 

o 

4J 

o 

XI 

•H 

cw 

o 

CO 

cu 

■a 

3 

0 

•H 

tu  cu 

o 

C 

<U 

a) 

oo 

CD 

o 

M 

to 

o 

o  • 

XI 

u 

4J 

<4H 

d 

1-1 

VI 

rH 

to 

4-J 

4-1  to 

u 

■H 

11) 

CO 

Ci 

u 

c 

00 

o 

5 

CU 

u 

Vl 

fl 

O 

o 

CO 

XI 

3 

MH 

-a 

O 

td 

4J 

a 

N 

VI 

00 

CD 

1-1 

C 

C 

e 

Vl 

J-> 

4J  cu 

td 

rH 

oo 

c« 

Q. 

CO 

O 

0 

u 

Vl 

a 

td  cu 

-H 

3 

d 

^ 

Vl 

O 

iH 

Vi 

o 

•H 

cu 

CU 

o  C 

4-1 

td 

a 

V4 

o 

CU 

a 

4-1 

XI 

o 

MH 

iH 

O.  Vl 

•a 

Vi 

X> 

■H 

d 

<J 

iH 

Xi 

o 

•H 

.-h 

CU 

C  d 

•H 

3 

«H 

4J 

OS 

Vi 

CJ 

to 

rH 

o 

a 

3 

c 

4) 

4J 

■H 

CO 

■a 

to 

O 

MH 

tj 

oo 

S 

a. 

T3 

N 

to 

CJ 

o 

CO 

td 

tu 

-a 

3 

rH  -a 

cd 

CJ 

CO 

X 

XI 

CO 

O 

rH 

•H 

td 

d 

o  3 

to 

1-t 

X! 

CO 

Vi 

Vi 

to  • 

4-1 

VI 

CU 

X 

td  tu 

o 

■H 

4-1 

-a 

xi 

■H 

td 

O 

cu 

■H 

o 

o 

> 

H 1 

cd 

iH 

<4H 

4J 

01  c 

•H 

rH 

CD 

cu  o 

09 

VI 

rH 

■H 

CD 

CD 

a 

4-1 

11) 

X) 

4J 

4-1 

XI 

iH 

■H 

cu 

*a 

CO 

•H  O 

O.  rH 

rH 

» — ' 

>  o 

•H 

CD 

a. 

s 

cu 

-H 

X 

o 

•H 

cu 

o 

CO  i-l 

■u 

X! 

•a 

XI 

CJ  iH 

CO 

CO 

3 

0)  Vl 

to 

4J 

I4H 

C4-4 

3 

CU 

CO  • 

XI 

cu 

4-> 

c  tu 

X! 

a 

3 

d 

4-1 

CU 

CO 

C  4J 

o 

o 

Oi 

to 

vi  a. 

o 

Vl 

!-l 

C 

-o 

•H 

CO 

CO 

CO 

OS 

CO 

Vl 

rH 

cd  a 

CJ 

iH 

cO 

rH 

d 

td  td 

XI 

iH 

4J 

r**  CJ 

e 

o 

O 

0) 

cu 

VI 

4-1 

a 

o 

Cu  (3 

iH 

Vl 

3 

■H  -H 

3 

^ 

c 

CU 

00 

■H 

i-i 

o 

c 

CD 

4-1 

O  -H 

rH 

o 

O 

O  4-1 

to 

3 

cu 

00  -H 

g 

CO 

CU  iH 

f-> 

CU 

XI 

td 

rH  C 

td 

00 

CD 

4-> 

Vl 

u 

CU 

■u 

CO 

rH  VI 

o 

•H 

■H 

CU 

4J 

d 

Vl 

Vl  MH 

UH 

a 

Vl 

rH  Cd 

iH 

c 

•H 

CO 

O 

u 

-o 

CO 

4J 

0)  tl) 

td 

tu 

co  td 

x: 

iH 

VI 

i-i 

CU 

•d 

cu 

cu 

O  iH 

•H 

a 

4-1 

td 

a 

CO 

O-i 

CO 

■H 

O 

CO 

rQ 

>  -a 

4-1 

Vi 

>W 

>1 

o 

> 

-o 

d 

cu 

4-1 

> 

co  vi 

4-1 

cu 

o 

CO 

3  to 

to 

o 

v-l 

x;  c 

td 

CU 

o 

o 

to 

CO 

cu 

•H  tU 

O 

X 

•H 

X 

CO  iH 

a  o 

w 

cw 

T3 

a.  -H 

tu 

oi 

c 

o 

3 

-a 

CD 

T3  > 

z 

4J 

4J 

cd 

3  to 

rJ 

U 
W 

CJ 

rJ 

o 

z 
o 
o 


►J 

2h 

O 
OS 
CO 
Oh 


rH 

CO 

cd 

T3 

1 

Vl 

CU 

Vl 

3 

4J 

O 

"3 

CO 

o 

Vl 

cu 

o 

T3 

tu 

0 

CJ 

4-1 

a. 

VI 

nf 

r  o 

to 

3 

CO 

rH 

•H 

a. 

d 

•H 

CO 

n) 

d 

CO 

r^ 

CJ 

Vl 

•H 

CO 

O 

03 

iH 

td 

c 

CJ 

c 

Vl 

TJ 

tu 

CO 

•H 

cu 

o 

CU 

rH 

to 

4J 

td 

Vl 

e 

o 

rH 

>i 

co 

iH 

Vl 

G 

td 

.cr 

o 

TJ 

CU 

r*. 

3 

CJ 

O.  iH 

4-1 

iH 

TJ 

00  rH 

iH 

Vi 

V4 

MH 

P 

d 

td 

rH 

o 

0) 

O 

•H 

iH 

u 

cd 

rH 

3 

3 

00 

o 

tU 

r*» 

Vl 

•o 

tr 

d 

CO 

Vl 

rH 

o 

<u 

1-4 

Vl 

3 

Vi 

o 

Vi 

to 

d 

o 

rH 

CO 

o 

o 

CO 

o 

Vl 

-H 

CD 

cu 

!-! 

o 

iH 

iH 

V4 

CO 

r-l 

X 

s 

4-1 

tu 

CJ 

4J 

01 

X 

4-4 

OJ 

00  X 

r^ 

d 

4J 

rH 

-H 

Vl 

4-1 

d 

CO 

CO 

-H 

tu 

T3 

rH 

CX 

CO 

CJ 

3 

si 

o* 

CO 

•a 

o 

T5 

c 

d 

d 

V4 

CO 

cO 

00 

O 

iH 

td 

O 

O 

00  *H 

CU 

■H 

d 

•a 

Vl 

to 

•H 

4-1 

00  rH 

X 

td 

C 

CO 

Cu 

CJ 

•H 

CJ 

■H 

^1 

•H 

cu 

-o 

Vl 

XI 

u 

d 

o 

cu 

< 

*H 

3 

a 

H 
CJ 

< 

% 

Z 

o 

rH 
H 

CJ 
Z 

Cl4 


d 

o 

XI 

iH 

rH 

d 

4-1 

td 

CO 

cd 

a. 

XI 

o 

iH 

CD 

CU 

0) 

iH 

CJ 

■H 

Vi 

3 

CW 

C 

CO 

3 

d 

iH 

iH 

o 

XI 

iH 

4-J 

V 

d 

CU 

4-1 

d 

01 

o 

d 

<u 

cfl 

o 

o 

X) 

U-l 

•H 

Vl 

o 

M 

O 

O 

0^ 

a 

W 

u 
o 
as 
cu 

■J 
O 


z 
o 
c_> 


►J 

cr 




i— 

O 

o 

i 

XI 
cD 

3, 

<r 

4-1 

4J 

01 

>, 

4-1 

CD 

CU 

> 

o 

-o 

c 

1 

o 

1-1 

CO 

l-l 

M-l 

1 

c 

rH 

X 

1 

rH 

CU 

o 

o 

IS 

C 

id 

•a 

i-l 

s 

CO 

4) 

l-l 

•H 

41 

-H 

rH 

rH 

•i 

0) 

o 

1 

H 

CJ 

01 

XI 

■H 

4-1 

41 

cO 

c 

M 

00 

a. 

• 

x 

CU 

CO 

,c 

4J 

rH 

01 

•a 

a 

00  T-l 

O 

•H 

o 

XI 

41 

d 

l-l 

X 

u 

CO 

CO 

id 

CX 

B 

AJ 

M-l 

o 

H 

C 

CO 

a 

41 

T-l 

XI 

4-1 

u 

C 

4-1 

d 

r-l 

d 

rH 

o 

o 

rH 

cu 

o 

o 

a 

rH 

•rl 

4-1 

id 

5 

1) 

c 

0) 

a 

c 

a 

3 

qi 

(X  T-l 

• 

CO 

rH 

o 

d 

CO 

CJ 

cd 

CO 

> 

^  s 

C 

aj 

CO 

CO 

CO 

tH 

CO 

o 

>4-l 

1-1 

CP 

X) 

•H 

o 

i-i 

co 

T-] 

<4-l 

CO 

XI 

o 

CO 

o 

cy 

01 

CJ 

cd 

ON 

30 

CO 

O 

O 

0) 

X 

CO 

U-l 

o 

tx 

CO 

• 

•H 

00 

CO 

o 

0) 

CO 

0) 

a, 

CU 

1-1 

U 

I-i 

1-4 

u 

0) 

T-l 

-C 

1 

u 

at 

o 

-H 

a 

CO 

tH 

X 

at 

-H 

CO 

1-4 

• 

M-l 

T-l 

0) 

O 

•H 

X) 

Ol 

CO 

c« 

01 

CU 

3 

X 

XI 

rH 

cj 

•h 

C 

T3 

CO 

o 

X 

o 

C 

1-1 

>>.X! 

O 

CU 

14 

u 

CJ 

XI 

M-l 

X 

4J 

3 

41 

Ol 

a. 

CO 

X) 

rH 

01 

XI 

rH 

id 

•a 

so 

CU 

cd 

c 

1-1 

u 

o 

rH 

C 

4J 

o 

o. 

4) 

41 

c 

a 

4J 

a 

CD 

rH 

4J 

3 

41 

CO 

XI 

a 

rH 

ij 

cd 

B 

■H 

00 

JS 

O 

-H 

CO 

I 

rH 

•H 

XI 

(X 

ecl 

o 

•H 

-r-l 

a> 

o 

at 

CO 

CJ 

CJ 

r-l 

3 

u 

oo 

■H 

13 

CO 

3 

4-1 

CO 

CO 

B 

4-1 

CO 

o 

a. 

4J 

41 

01 

M 

XJ 

XI 

o 

o 

T-l 

3 

01 

o 

10 

c 

XI 

01 

■H 

C 

CD 

CO 

00 

o 

a. 

l-l 

S 

CJ 

a 

CJ 

C 

41 

0) 

> 

to 

^4 

at 

4) 

i-i 

4-4 

U 

XI 

O 

u 

3 

X 

^ 

X 

rH 

Q 

■H 

e 

■H 

CJ 

c 

a; 

•H 

4-1 

o 

cd 

T-l 

41 

V-i 

M 

14 

r-l 

CO 

CO 

X 

M-l 

o 

IX 

CU 

CJ 

•H 

CD 

c 

i — 1 

4-1 

cd 

u 

•H 

•H 

c/o 

o 

CO 

M-l 

1-4 

l-l 

rH 

41 

o 

at 

O 

o 

CJ 

4J 

CO 

o 

CJ 

i-H 

0 

4J 

rH 

th 

i — 1 

CL 

4) 

B 

o 

10 

M-l 

XI 

CU 

c 

4-1 

0 

cd 

to 

41 

B 

CO 

0 

c 

XI 

0) 

CO 

XI 

r»> 

d 

cd 

rH 

B 

a) 

d 

-H 

p 

<4-l 

a) 

•H 

O 

•H 

O 

Ol 

CO 

4-1 

3 

T-l 

a, 

0) 

O 

o 

rH 

4J 

l-i 

4J 

d 

01 

O 

Ol 

4-J 

01 

U 

•H 

o 

H 

o 

l-i 

o 

cd 

Vl 

o 

X 

J3 

t4 

XI 

CD 

XI 

CL, 

4-1 

XI 

1-4 

4J 

•H 

3 

CJ 

T-l 

cd 

X 

CO 

t4 

C 

0) 

4J 

d 

4) 

u 

01 

a- 

4J 

CO 

C 

•H 

CO 

CO 

CO 

rH 

0) 

4-1 

o 

41 

r-l 

o. 

4-1 

u 

4) 

cx 

iH 

u 

3 

t-l 

-H 

rH 

0) 

> 

1*4 

01 

o 

> 

X! 

r*. 

X 

3 

U-l 

41 

■H 

X 

(X 

CO 

CO 

to 

3 

X 

to 

oo 

01 

O 

to 

1-4 

• 

C 

a. 

X 

o 

<o 

41 

B 

M-l 

01 

S 

-H 

c 

CO 

4-1 

o 

M-l 

> 

a 

CO 

CO 

CJ 

o 

01 

c 

CJ 

4-J 

o 

t! 

•a 

0) 

CL 

4) 

o 

B 

4-1 

at 

>4-l 

01 

X 

-H 

O 

X 

XI 

c 

cd 

14 

CJ 

1-1 

cfl 

T-l 

at 

T-l 

X 

CO 

o 

o 

X 

)-i 

4) 

O- 

M 

•H 

CO 

t-> 

4J 

l-l 

4J 

4-1 

4-1 

o 

G 

o 

o 

41 

X) 

X 

M-l 

co 

XI 

o 

at 

00 

O 

o 

4J 

fx 

0) 

3 

4J 

CO 

Ol 

CO 

c 

3 

D, 

cm 

T-l 

•H 

M 

l-l 

l-l 

Xi 

01 

rH 

co 

0.) 

4) 

a 

co 

c 

XJ 

4-1 

iH 

CO 

x 

XI 

CO 

XI 

X) 

>-. 

O 

O 

XI 

C/3 

4-1 

O 

l-l 

Xi 

5-4 

O 

rH 

XI 

CO 

•H 

1 

X 

00 

d 

u 

iH 

a 

II) 

l-i 

4J 

01 

cj 

id 

CD 

41 

rH 

4-1 

i-i 

0) 

Qj 

>. 

CO 

M-l 

0) 

CO 

O 

4-1 

rH 

3 

X 

4-1 

CO 

CO 

01 

CO 

CO 

rl 

3 

^ 

o 

-H 

CO 

xi 

rH 

CJ 

C 

XI 

J3 

o 

X 

CJ 

-H 

l-l 

d 

4-1 

«» 

•H 

1-1 

4J 

3 

u 

o 

0) 

>| 

o 

MH 

CO 

3 

CU 

O 

O 

o 

41 

41 

o 

41 

•H 

CO 

41 

o 

CO 

01 

3 

0) 

a 

r^ 

5>N 

XI 

CJ 

cd 

O 

iH 

X 

m-i 

u 

•H 

-H 

I-l 

u 

u 

3 

JS 

CJ 

rH 

CO 

M-l 

C 

X 

u 

14 

3 

a. 

a 

41 

rH 

X 

•H 

0) 

>,  pi 

o 

M 

• 

4-1 

■H 

a 

4-1 

a. 

XI 

4J 

-a 

41 

1-1 

o 

4-1 

D- 

X) 

o 

o 

N 

4J 

rH 

00 

l-l 

X 

00 

4-1 

P. 

at 

4J 

M 

■» 

CO 

id 

M-l 

o 

CO 

rH 

u 

CO 

r-l 

os 

u 

4) 

•H 

d 

XI 

CO 

t^o 

Cu 

01 

a 

d 

i-i 

4) 

oo 

4-1 

X3 

•H 

41 

4-1 

X. 

U 

T-l 

41 

at 

MH 

co 

CO 

0) 

oo 

(X 

l-l 

to 

OJ 

0) 

exx; 

CO 

rl 

4-1 

)-J 

id 

01 

> 

3 

0) 

(0 

C 

l-l 

VI 

at 

3 

01 

M-l 

> 

01 

l-l 

T-l 

o 

l-l 

Pi 

3 

CO 

3 

4-1 

■H 

(X 

01 

41 

cu 

41 

3 

XI 

D. 

o 

rH 

XI 

CO 

CO 

41 

3 

o 

ij 

T-l 

CO 

4-1 

XI 

X 

cr 

CJ 

CJ 

co  ji 
cd  H 

UH 

41 

XI 

CO 

H 

•H 

CO 

iH 

CO 

X 

> 

T3 

4) 

c 

rH 

1-1 

CO 

4-1 

X) 

41 

CO 

OJ 

o- 

01 

3 

C 

4-1 

41 

,£S 

4) 

0) 

CJ 

0) 

rH 

4-1 

a> 

CO 

0) 

CJ 

o 

rH 

(X 

4J 

0> 

•H 

01 

CO 

X 

X 

CJ 

B 

IH 

14 

-H 

X 

41 

CJ 

u 

CO 

O 

|H 

c 

1-1 

o 

iH 

i-i 

B 

01 

XI 

u 

CO 

CO 

TH 

CO 

0 

oo 

B 

IJ 

4-1 

4J 

o 

01 

U-t 

l-l 

CU 

o 

3 

■H 

3 

MH 

o- 

o 

l-l 

CO 

C 

o 

a. 

l-l 

(X 

XI 

■H 

c 

o 

M 

CO 

O. 

c 

X 

u 

t-l 

X) 

-o 

U 

X) 

CO 

s 

O 

o 

CO 

1-4 

CO 

0) 

o 

0 

T-l 

rH 

01 

1-4 

41 

ex  xi 

d 

01 

CD 

c 

XI 

O 

o 

(X 

a) 

ri 

(X 

0) 

X 

o 

M-l 

c 

a. 

(X 

X 

rH 

l-l 

u 

u 

CO 

X5 

IJ 

CD 

S-i 

rH 

•H 

41 

4-1 

l-l 

c 

3 

u 

o 

01 

o 

X 

CO 

00 

Ol 

CO 

o 

41 

41 

41 

41 

4J 

4J 

41 

C 

3 

o 

3 

d 

CO 

4J 

01 

o 

-a 

o 

cj 

01 

o 

iH 

41 

CO 

XI 

XI 

4) 

T-l 

4-1 

4-1 

XI 

a 

O 

CO 

ai 

O 

o 

41 

o 

T-l 

CJ 

CJ 

o 

rl 

d 

1-1 

0) 

X 

1-1 

X 

S-i 

mh 

X 

T-l 

o 

c 

a 

X 

41 

MH 

01 

l-l 

d 

M-l 

T-l 

c 

C/J 

X 

i-i 

l-i 

X 

l-l 

cd 

d 

41 

o 

O 

m 

c 

:-i 

4J 

Cu 

3 

o 

o 

H 

XI 

o 

cd 

-H 

4-1 

3 

•a 

< 

XI 

o 

T-l 

r-l 

4J 

01 

CO 

4-1 

CO 

4J 

3 

C4— 1 

T-l 

1-4 

41 
4-1 

41 

I-i  O 


d 

l-l 

0) 

41 

o 

X) 

iH 

o 

r— S 

s 

V4 

CJ 

M 

01 

D. 

O 

-H 

cd 

rl 

> 

XI 

rl 

0' 

rl 

MH 

CU 

rl 

CO 

O 

rH 

o 

01 

4-1 

CO 

4J 

rl 

en 

ur 

OJ 

rH 

B 

X3 

1-4 

CO 

d 

CU 

3 

CJ 

oo 

CJ 

rH 

•H 

•H 

O 

■H 

XI 

to 

u 

CO 

01 

to 

D. 

Pu 

B 

at 

H 
O 
< 

% 
$ 

z 
o 

M 

CJ 

z 


c 

o 

XI 

•H 

rH 

d 

4-1 

CO 

co 

CO 

a 

XI 

CJ 

•H 

CO 

01 

OJ 

iH 

u 

T-l 

rl 

3 

I4H 

d 

CO 

3 

C 

iH 

-H 

o 

xi 

■H 

4-1 

r4 

c 

4) 

4-1 

c 

P-l 

ofl 

CJ 

d 

41 

CO 

o 

o 

XI 

MH 

-H 

U 

o 

M 

O 

a 

(X 

I  XI 
41 

41  B 

O 

Ij  41 

3  M 

O  -rl 

CO  4J 

d 

CO  41 
4J 

CO  U 

xi  o 

41  - 

x:  u 

4J  41 
41 

CO  X  e~ 
•H    CO  XI 

u 

u    II  O 

CO  o  o 
X  aj  41 
5  MH  U 


Exhibit  II-4 


□ 


o 

LL 

s 
g 

r- 
< 

o 


CC 

LU 

> 

00 
1X1 
CC 

Q 

LU 

o 
o 

CC 
Q. 

Q 
2 
< 
GO 
UJ 

to 
O 
2 
O 
< 

O 

o 

CC 
C/5 
Q. 


3 
I 


□ 


< 
□ 

u 

X 

□ 


8 
i 


Reason  for 
Discrepancy 
•  • 

Reconciled 

Results 
(If  Neither 
Is  Correct) 

Check  which  is  Correct 

■ 

c 

•z  I 

Q 

Neither 

5 
< 
■ 

CC 

Abstract 

Abstract 
Code 

Re-abstract 
Code 

Data 
Source 

Write  out  all  diagnoses  as  they  appear  in  medical  record 

Designate 
A-  Admitting 
P-  Principal 

Reason  for 
Discrepancy 

Reconciled 

Results 
(If  Neither 
Is  Correct) 

Check  which  is  Correct 

Can't 
Determine 

Neither 

Re-abtt. 

Abstract 

Abstract 
Code 

Re- 
abstract 
Code 

Data 
Source 

Write  out  procedures  as  they  appear  in  medical  record 

Designate 
P  for 
Principal 

c  ~  r  o  c 

its-- 

5  6  a  c  2 
S  •=  5  |  | 

"  i  i  :  I 

1  I  £  2  2 

®  ST   _  —  *» 


'5  9  m 
to    «-  ^ 


I  2 

c  o 

S  3 

"o  o 


•O  ■ 

i  ' 
si 


£ 

o 
o 


I] 


«  4 


S  £  a  a 

s  5  ■  s  = 

o  «  •  c  >  "5 

a  -s  J  ||  | 

q  s  a  1 1  | 

(3    ™  T3  o  aw 

<3  ii  it   ii   ii  ii 

a  u-  Q  O  a.  O 


s.   "s  "s  e 

ST       9?   «  ® 


i  -5  £ 

■  ■  s 


■»»    *■     U  w 


c  c  c  c 


Q.  —  2.  o  » 
o  e  b  «  ' 
c  =   c   -  = 


"5  "S  ~°  "2  "o 
O  O  O  O  O 

>-  (N  P)   V  10 


•  T3  £  a 

r  c3  5  a 


Id 
Ptf 

Q 
UJ 

u 
o 

OS 

p* 

o 

0! 
H 

z 
o 


►J 


TJ  <M 

U  0) 

O  *J 

O  10 

o 

tj 

3  <u 

CU  i-l 

•H  <M 

>  -H 

a)  iH 

m  <a 

3 

o  o* 


CO  u 
P->  cu 
X! 
0)  4J 
J3  O 

u 

M 

4J  O 
CO  w 

*J  M 

O 

41  CD 

r-l  i-l 
X>  > 
•H  W 

co  41 

CO  o. 
O  3 
O,  to 

co  u 

•H  O 
4J 

u  eg 
H  C 


4J 

O  II  £ 

C  X!  O 
4J  3 

rH  CO 

i-H  <M 

■H  O 
3 

G  41 

l-i  O  l-J 

O  i-l  3 

U  U  T) 

o  o  <u 

UJ  <U  CJ 

U  H  O 

u  1)  M 

(0  CD  pi 

■a 

eg  <u 


u 
O 

■-I  Xl  

CO  i-l  TJ 

C  at  c 

•H  CD  CO 
00  O 

i-i  a  to 

U  "H 

0  4J  CD 
CO  O 

cu  «i  a 

X  XI  M 
4j  a) 
01  i-( 
TJ  XI  TJ 
C  u 
<a  i-l 
ti  a) 
- — *  O 

l-l  1-1 

CD   CU  CJ 

j3  ai  a 

Sl-ii-l 

01  OO  M 

0  cu  a. 


>»  • 
X>  0 
OJ 

TJ  1H 

<u  u 

r-H  1-1 

tJ  CD 

C  >• 

tO  JS 

X.  o. 

4-1  00 

to  a 

41  iH 
J3  TJ 

c 

5n  4> 


41  (0 
M 

41  41 

C  XS 

41  u 
ao 

x: 

4)  4-t 

I-i  -H 
a)  3 

co  oo 

c  a 

O  -H 
iH  U 
4-1  U 

CO  41 

3  M-l 

*->  C 
■H  O 
CD  CJ 


a 

o 

•H  14-1 

4J  O 
aS 

U  41 


4) 
M 

O  iH 
OS  4J 

w  a 

CU  41 


iH  p,  41  1-1  O 
u-i  S  ■ 


u 

CU  4) 
f»  J3 

4-1 

M 

41  <4-l 

tj  o 

M 

O  CD 
CD 

41  41 

U  H 

3  TJ 

"O  )-i 

41  CO 

CJ  00 

O  41 

a 

01 

■a  a 
a  o 

CO  T) 

CD  41 

•H  Xl 
CD 

O  TJ 

C  rH 

00  3 
CO  O 

1  -£ 
a  co 


'"J 

J* 

oo 

CO 

*~ 

a 

I-i 

rn 

~T 

i-i 

O 

, 

 " 

u 

■u 

o 

'  I 

3) 

ca 

.-t 

c 

«r 

Q.  1-4 

J- 

•V 

0 

T3 

4-J 

CO 

o 

M 

w 

—J 

o 

O 

O 

OS) 

O 

c 

u 

o 

" 

J 

or; 

o 

c/J 

4-1 

CJ 

0) 

Pu 

CO 

<y 

cO 

p 

c 

-H 

I-s 

41 

1-1 

> 

0 

x; 

•o 

m 

c 

o 

4J 

14 

t-) 

CO 

o 

"o 

4-1 

o 

rH 

C 

CJ 

Q 

4) 

41 

u 

Ki 

4-1 

XJ 

0 

s 

«H 

CO 

4) 

41 

O. 

4-J 

C 

CU 

00  -H 

P 

«H 

00 

c 

> 

o 

cy 

o 

CO 

11 

U 

4-1 

l-l 

VI 

ao 

o 

u 

C 

o 

l-l 

<d 

TJ 

4J 

■H 

o 

41 

41 

O 

4J 

c 

» 

>. 

03 

OJ 

41 

O 

M 

o 

■H 

4J 

a.  -h 

CO 

> 

CD 

0 

> 

V4 

41 

& 

XI 

O 

4) 

M 

Q) 

ca 

o 

a. 

>>  CO 
4J  41 

Tt  U 

•-t  3 
CO  T) 
3  41 

cr  o 
o 

a)  m 

4->  O. 

CO 

TJ  u 
iH 
00  TJ 

a  3 

iH  CO 

o 

00  CJ 

a  i-i 

o  TJ 

o 

41  i-l 

>  M 

O  41 

Xl  CU 
CO 

41  to 

x:  4i 

4->  l-l 

3 

O  TJ 
4J  U 

u 
c  o 

O  l-l 
iH  O. 


TJ  O 

TJ  W 

CO  4-1 

c  § 

M  CJ 


a.  u 

CO  i-l 

Xi  TJ 
O  3 
a) 

41 

41  ra 

CO  4-1 

•g 

TJ  >w 
41  O 

o  a 

iH  O 
O.  i-l 

a  u 

41  O. 

•H 
41  H 
XI  O 
CO 

O  41 
CD  TJ 
i-l 

tO  TJ 
41 

TJ  i— I 
■-I  i-l 
3  td 

O  4J 

*  i! 

CD  TJ 


O 
i-l 
I 


H 
CO 
M 
l-J 

o 
w 

o 

l-l 
o 

8 

z 
o 
u 


I 

c 

o 
co 

CU 


4J 

a 

41 

O 

fi 

uf 

CO 

c 

l-l 

•H 

M 

O 

CO 

41 

4-1 

01 

0) 

00 

^-1 

M 

D. 

B) 

E 

x: 

O 

o 

U 

CO 

C 

■H 

l-l 

TJ 

00 

c 


CU 
XI 


01 

to 

01 

CO 

•H 

mmar 

accu 

the 

3 

►> 

(0 

an 

X 

41 

u 

oo 

c 

01 

i-i 

c 

C-» 

ro 

c 

0) 

X 

-o 

(3 

o 

Ol 

i 

ca 

CD 

4-1 

■H 

•H 

01 

>, 

O 

-o 

^4 

i-l 

Pa 

41 

CD 

o> 

0 

0 

u 

o 

-H 

x: 

< 

o 

4J 

a, 

H 
O 
< 

< 
•-1 


d 

o 

-H 

T3 

4J 

C 

TO 

a) 

n3 

-—V 

CJ 

aj 

T3 

•H 

■H 

to 

Ol 

Ol 

CU 

O 

•H 

l-l 

3 

1-1 

c 

CO 

3 

C 

4J 

•H 

o 

-o 

■~! 

c 

l-l 

c 

Ol 

4J 

0) 

00 

o 

C 

TJ 

CO 

o 

o 

l-l 

C4-I 

1-i 

l-l 

CJ 

o 

Q 

Pu 

s  

•J 
< 
z 
o 

H 

o 
z 


(I 


co 
w 
PS 

S3 

a 

CU 

o 
o 
a; 
p* 

►-3 

O 

o 
o 


1-1 
< 


I 

X  • 

ai  o 

to  z 


00  1-1 

c  a 

iH  00 

44  B 

i-l  to 

•a  m 

a)  h 

CO  c 


•a 
c 

s> 

<u  00 

Vi    C3  CO 

tj  a  co 

CD   O  O 

CJ    CJ  B 

O  43  OC 

U  4J  CO 

PL  Vi  iH 

O  Q 
X)  MH 
C  MJ 

co  CO  o 


■H  i-l  TJ    CD  <! 
H  rl 


Vl  Tj 

O  0) 

00  H 

00  iH 

01  CO 
O  4J 

o  <u 

U  TJ 
Cu 

00 

O  CO 

a! 

to  en 
cl,  11 

TJ 
44  O 

CO  cj 
43 


00  cu 

o  e_> 
c 

00  a 

CO  O 

•H  VI 

TJ  m-i 


44  CJ 


i-l    B  -H 


0) 

u 

3  O 
00  CD 
00  CL 


tj  cr 

S  3 

CD 

0.  c 

Cu  O 
<  -H 
44 

CO 


01 

l-l 

43 
CO 

cfl 

■H  <U 
Vt  43 
CD 
44 

u  i 

o 

o 

CD  OS 
B  to 

O  Oi 
CO 

co 

CO 

0)  43 

TJ  CJ 

■H  iH 

>  -s 

o  3 
u 

0.  00 

0J 

:  to 
-  o 
00  G 
c  00 

i-l  CO 
TJ  -H 
O  t) 

o 

44 

Vi  00 


-H  00 

u  <u 

0J  TJ 

cu  o 
00  o 


•H  00 

tj  b 
cu  a) 

01 

00  u 
u  cj 

■H  CO 

O  *H 

U  CO 

C  3 

•H  (3 

CU  § 
44 

CO  M 

M  O 
O 

Cu  TJ 

U  CD 

O  N 

CJ  1-4 

S  M 

iH  CD 
44 

O  3 

44  CU 


00 
b 

iH 

o 
0 

00 
(3 
1-4. 
Vl 
O 


00 

a  co 

t(    CO  M 

CO  rH  CO 

44  CO  O. 
B  44 

O  -H  fl 
CJ   Cu  B 

CO  CO 
CD  O 

44  43  Cfl 

VI  -H 
OHM 

Cu  CO  CD 

CD    3  44 

Vi  TJ  t4 


U 

o 

Cu 
CD 
VI 

o 

43   09  44 
i-l 

CO  00 
>>  44 
M 


VI    >  CJ 

3  "H 

cu-a  "O 

44   13  4) 

3  -H  B 

O  O 

8  "H 

M    O  44 

CD    VI  C 


3 

Cl  CO 

a  *j 

O  CO 
CJ  -O 


CD  13 

>  0) 

O  -H 

43  44 

CO  CO 


VI 

CJ  . 

Cu  CJ 

O  CO 

rH  Vi 

CD  3 

>  O 

CD  CJ 

a  co 


0  CO 

OS-  fv, 

co  vi  «) 

PL,   CD  43 

44  44  • 

3    VI  00 

CD   CO  00  Vi 

■H   3  CS  CD 

>    Q*  lH  44 

CD  00  CD 

01  co  3  a 


M 
TJ  Cu 

§  2 
Cu 
00  O. 
00  CO 

s 

•H  44 
Cu  CO 
3  O 

o  a 

VI 

00  CD 
43 


44  CD 

CD  43 
O 

S  TJ 

00  »— t 

CO  3 

i-l  O 

TJ  3 


00 

5 
5 

CD 

a 

CO 

44  • 

CO  CO 

CD  44 

00  VI 

00  O 

3  Cu 

CD  CD 
V< 

< 

44 

X  3 

T*  CU 

TJ  44 

B  3 

CD  O 
Cu 

Cu  43 

<  » 

-  CO 

B 

iH  00 

co  B 

00  -H 

CO  M 

•  -  l-l 

CD  CO 

N  B 
>>  CO 
l-l 

CO  VI 

(3  O 

CO  M-l 


43 
CJ 
1-1 

43  • 

»-s 
00  44 

a  -n 

M  Cu 

V  CO 

44  O 
44  43 

CO 

Cu  iH 
CO 

Vi  3 
O  TJ 
Vi  -H 
Vi  > 
CD  i-l 
TJ 

-  s 

CD  i-l 
■-I 

43  >■> 
i-l  43 
00 

CO  i-l 

O  -H 

a  co 

M 

<4-l  0J 
1-1  > 

O 

CD  44 

(3  00 

■H  i-l 

B  X 

U  CD 

CD 

44  Ss 

s  1 


CO 

a 

VI  CD 

CD  CO 

44  CD 

44  43 
CO  H 
CU 


CO 

44 

o 

CO 
Vi 
vl 
co 

43 
CO 

M-l 

o 


Vi  — . 

O  Vl 

Vi  O 

Vi  44 

CD  CO 
C 

Vi  i-l 

O  TJ 

O 

00  O 
13  CJ 
i-l 

TJ  Vi 

O  O 
CJ  ^ 


O  CO 
44 

CD  i-l 

>  Cu 

i-l  00 

44  O 

CO  43 
44 

C3  43 

CD  CJ 

CO  to 

CD  CD 
Vl 

Cu  Vi 

CD  O 

Vi  <4-l 


Vl  CD 

0  Vi 
44  3 

TJ  TJ 

rt   CD  (3 

5    O  CO 

CO  O 

Vl   Vl  CO 

TJ   Cu  i-l 
CO 

01  1— I  o 
1— I  O  13 
Cu  Vi  00 
B  44  CO 
CO   C  i-l 

co  o  a 

CJ 

CD  iH 

I   >>  CO 

CO    44  Cu 

CD  i-l  i-l 

■-I  CJ 

CD   CO  (3 

43  3  i-l  s-* 

44  cr  vi 

P-4 

CD  CO 
43  44 

CO  O 
SSTJ 

H  G  i-H 

43   00  O  CD 

i-l   3  iH  CO 

CD    O  44 

CD  i-l  CO 

o  >  o 

Cu  CD  i-l 

vi  u-i 

TJ    CU  i-l 


o 
rt 
vi 

CJ 

01 


3  CD  CD 
O  43  > 


M 
3 
T3 

« 

CJ 

O 
Vi 


CJ    44  '  CV| 


>. 

OO 

TJ 

rH 

G 

u 

Vl 

T3 

0) 

43 

Vl 

CD 

rt 

43 

0 

CO 

0) 

43 

CO 

0 

43 

TJ 

1 

rt 

10R 

4-1 

ao 

1 — 1 

44 

1 

i-i 

Vl 

rt 

C 

4-1 

O 

CU 

cd 

(3 

>. 

(0 

O 

T3 

CO 

0 

M 

0 

CJ 

O 

Vi 

O 

U-l 

■H 

l-l 

a 

Vi 

O 

Vi 

CJ 

ft. 

CJ 

co 

CJ 

CD 

44 

CO 

rt 

2 

CO 

O 

^1 

Vi 

rt 

(0 

CO 

43 

O 

•H 

CO 

CO 

1-1 

43 

cu 

0 

TJ 

w 

Vl 

■H 

01 

-U 

rH 

Vl 

CO 

& 

TJ 

0) 

0) 

C 

43 

0) 

O 

CD 

CO 

Cu 

CU 

3 

0 

O 

00 

C 

TJ 

VI 

O 

>,  44 

3 

§ 

OF 

14 

(A 

M-l 

O 

43 

43 

Vl 

44' 

01 

i-l 

TJ 

Vi 

CO 

TJ 

CO 

CD 

Vi 

3 

Vl 

a) 

CO 

VI 

M-l  i-l 

a 

Vl 

CO 

44 

43 

a 

C 

C 

CD 

CO 

CU 

43 

44 

cu 

>>  -H  TJ 

CO 

a 

O 

rt 

43 

_o 

i-l 

CO 

O 

4-1 

ICES 

Vl 

at 

a. 

44 

CD  O 

CO 

a} 

O 

CO 

00 

<a 

Vl 

rt 

CD 

00 

01 

CD 

rt 

CU 

00  1-1 

CO  CJ 

CU 

O 

CO 

CD 

0 

4J 

rH 

3 

a. 

43 

CU 

CO 

43 

S 

rH 

cO 

CD 

00 

Vl 

C 

1 

CO 

4) 

0) 

VI 

■H 

I" 

■H 

0 

CJ 

4-1 

4-1 

rt 

rt 

rH  CD 

43 

c 

00 

a) 

43 

> 

VI 

CO 

TJ 

Vl 

3 

0 

01 

0 

TJ 

43 

44 

rt 

rH 

CO 

44 

rt 

CD 

0 

O 

O 

O 

Cl 

CD 

c 

43 

44 

CO 

O 

CO 

44 

CO 

iH 

4-1 

TJ 

4-1 

Cl 

O 

to 

O 

a 

44 

O 

rH 

O 

3 

<9 

CO 

O 

TJ 

e 

01 

TJ 

CO 

Q0 

CO 

c 

O 

iH 

01 

c 

CO 

•H 

rH 

> 

■H 

rt 

01 

•H 

G 

G 

C 

c 

rt 

4-) 

i-l 

09 

43 

>. 

rH 

i-H  • 

CO 

rH 

TJ 

<d 

VI 

M-l 

CO 

U 

rt 

CO 

01 

CO 

>> 

4i 

44 

1— 1 

XI 

44 

<d 

>. 

CO  00 

43 

CU 

VI 

•H 

cu 

TJ 

Vl 

l-l 

Vi 

43 

O 

a 

c 

0 

44 

43 

CJ  iH 

TJ 

a 

3 

CO 

44 

O 

4-1 

CO 

3 

O 

T3 

0 

CO 

0 

GS 

rt  CO 

4-1 

Vl 

cu 

O 

CO 

rH 

CO 

O 

CD 

CO 

H 

S3 

TJ 

-O 

43 

CO 

43 

CU 

TJ 

U-l  0 

O 

0 

CU 

rH 

01 

CO 

"cu 

CO 

rH 

u 

CU 

T3 

H 

0) 

0) 

CU 

00 

a 

CU 

rt  C 

(3 

3 

43 

u 

rH 

44 

CO 

Vl 

01 

B 

Vl 

O 

<B 

CU 

Vl 

a 

cu 

CU 

00 

CD 

O  00 

1 

4-1 

I" 

CJ 

3 

Ss 

3 

01 

CO 

O 

H 

O 

3 

-o 

■H 

TJ 

TJ 

TJ 

3 

CD  CO 

CO 

0 

rH 

G 

43 

1 

CO 

O 

O 

Vi 

CO 

0 

O 

CD 

c 

O 

3 

CO 

Cu  -H 

|> 

0 

B 

K 

CO 

CU 

CO 

C 

4-1 

3 

cu 

CU 

Pi 

P-. 

O 

0 

4-1 

3 

1-1 

O 

»T) 

0 

00  TJ 

m 

rt 

CD 

CO 

43 

O 

O 

13 

rH 

Vl 

ete 

CU 

jS 

F-l 

CD 

01 

44 

01 

43 

44 

C3l 

o. 

TJ 

TJ 

CJ 

CO 

43 

TJ 

44 

G 

Vl 

a 

CO 

OI 

OI 

O 

CU 

c 

1 

CO 

e>« 

43 

4-1 

01 

3 

■H 

3 

0 

44 

4-1 

CD 

O 

Vl 

CO 

rH 

CO 

3 

G 

0 

>» 

00 

O 

0 

O 

CO 

9 

3 

G 

CJ 

Cl 

CO 

M-4 

iH 

43 

■4-1 

CU 

M-l 

O 

-0 

TJ 

0 

4-1 

C 

TJ 

0) 

1-1 

0) 

CD  TJ 

O 

CO 

00 

rt 

43 

CO 

43 

1 

c 

00 

44 

O 

•H 

CD 

<u 

CD 

0 

Vi  B 

VI 

TJ 

c 

44 

CD 

rH 

0 

n) 

G 

B 

CJ 

■s 

rt 

44 

0 

CO  CO 

00 

4-> 

C 

1-1 

0 

co 

43 

CO 

CU 

4-1 

•H 

TJ 

TJ 

0 

CO 

CO 

VI 

G 

CO 

G 

C 

Vl 

4J 

3 

TJ 

CJ 

01 

OI 

CO 

CD 

Vl 

TJ 

CO 

44 

Cu 

CO  00 

rt 

VI 

rt 

cu 

e 

O 

CL 

rt 

43 

co 

cu 

44 

Cu 

<U 

(0 

CD  B 

G 

01 

rH 

CO 

rH 

TJ 

0 

CJ 

3 

44 

3 

CO 

CO 

TJ 

rH 

ao 

3 

CD  -H 

c— 

rt 

43 

0) 

n 

CO 

O 

44 

1 

0- 

OI 

CO 

0 

rt 

CO 

Vl 

0 

c— 

G 

0 

5NTJ 

T3 

c0 

4-> 

c 

44 

44 

O 

0) 

Vi 

CO 

43 

cu 

CO 

B 

rH 

rt 

0 

TJ 

-H 

rH 

O  O 

CO 

Vi 

M 

G 

1 

rt 

CD 

O) 

4-1 

11 

44 

CO 

X 

1-1 

50 

43 

CJ 

Vl 

CU 

TJ 

<4H 

rH  CJ 

O 

4-1 

3 

O 

OI 

Cu 

3 

G 

Vl 

at 

Cu 

O 

43 

0) 

CO 

t>» 

44 

CU 

0 

to 

O 

Cu 

rH 

M-l 

CO 

Vl 

CO 

0) 

0- 

O 

O- 

3 

-0 

O 

O 

Cu  TJ 

TJ 

rt 

rH 

c— 

VI 

3 

O 

44 

a  0 

4=: 

CU 

Vl 

c— 

0 

B 

00  TJ 

rH 

TJ 

1 

u 

TJ 

43 

rH 

B 

<>• 

O 

TJ 

rH 

CO 

TJ 

CO 

VI 

CD  44 

M 

43 

CO 

CU 

CO 

0) 

43 

G 

CU 

01 

0 

Cu 

CD 

CO 

rt 

S>*43 

c0 

rH 

0) 

CO 

44 

0) 

CO 

O 

0 

4-1 

Vl 

Cu 

cu 

0 

C 

rt 

00 

> 

0 

4J 

CD 

0) 

rH 

44 

>, 

O 

4J 

0) 

0 

VI 

01 

3 

>1  TJ 

3 

cu 

0 

a) 

CU 

cu 

TJ 

G 

01 

0 

cu 

3 

TJ 

CU 

14 

4J 

CU 

14-1 

iH 

c 

44 

O 

CO 

0 

B  CD 

CO 

TJ 

S 

TJ 

rt 

43 

43 

O 

•H 

rt 

Vi 

Cu 

43 

O 

43 

CO 

CJ 

a 

rt 

U-l 

CU 

CO 

43 

TJ 

SI 

co  B 

0) 

rt 

CO 

O 

Cu 

44 

3 

O 

TJ 

CU 

3 

4-> 

00 

O 

4-1 

rH 

CU 

CO 

rt 

CO 

a 

TJ 

CO 

43 

a  0043 

M 

O 

c/5 

G 

O 

44 

G 

3 

Vl 

44 

CO 

O 

3 

01 

B 

B 

44 

1-1 

4-1 

4-1 

01 

CU 

0) 

CO 

O 

O 

O 

iH 

OI 

TJ 

CU 

-H 

01 

OI 

43 

Vl 

CO 

c0 

cu 

0 

CJ 

CU 

TJ 

O 

CD 

3  00 

CO 

TJ 

OI 

44  44 

CD 

OS 

OS 

00 

Vl 

B 

Vl 

OI 

Vl 

Vi 

CO 

0 

rH 

CU 

0 

O 

Vi 

O 

rt 

G 

O  CO 

CD 

O 

Vl 

M-l 

CO 

0 

CO 

to 

CO 

1-i 

<c 

CO 

< 

43 

< 

< 

44 

0 

0 

CD 

43 

TJ 

< 

CJ 

4-1 

iH 

33  CO 

rt 

O 

< 

43 

0 

44 

Q 

Ph 

Oh 

rH 

TJ 

0 


3 

S3 
O 
M 
Eh 
O 
S= 


00 

*H 

TJ 

00 

CD 

C 

0 

CU 

CO 

c 

Vl 

ao 

3 

C^l 

co 

TJ 

1 

■H 

OI 

< 

a 

OC 

CD 

oc 

Vl 

H 

CO 

C 

0- 

1 

1 

•H 

33 

< 

TJ 

TJ 

Q 

O 

G 

c 

cj 

O 

CO 

rt 

IH 

CO 
US 

os 

3 
Q 

w 
u 
o 

Pi 
P* 

J 

o 
ps 
h 
z 
o 


H 
CO 
i-l 
.J 

cj 
fc-1 
3fi 
u 

►J 

o 


z 
o 
o 


i-3 
< 
3 

C 

ES 
<: 
o 

o 
os 

CO 


00 

e 

tj  a  ess 

«  u  a 
a.  <u 

0  *— I  i-l 

i-l  H  !-' 

3  a) 
ana. 

o  OD 

cn  u 
pu  a  o 

4)  4) 
4»  .C  H 
Si  u  41 

4J  CO 

jit 

IH    CO  CD 

O   rj  5 
4-1 

CO  T3 

I   C  Vi 

eg   a)  o 


at  o 

4J  u  H 

CO         O  TJ 

iw  i-l   CO   41  M-l  i-H 

o     u  >  a 

CD  4J  <D  ot  o 

CO    CO    CO  ft  r-\  3 

a>  eg  J3  -rl 

i-l  o  k  h  w  o 

q.  r-i  es 

o  <a  C  i-f  &>  c/s 


CU 

>  41 
CO  .O 


o  so 
e 


O  VI 
4J  o 


co  a 
u  u  p 


tt-H  in 

CO  CO  *0 
O  -H  l-i 

x:  >  o 

CJ 

11  00  41 
.C  C  M 
4J  -rl 

Vl  i-l 
>.  O  CO 


1 
o 

■a  o 

CO 

J5 
CJ 

■H  CO 

a  'M 
O  vj 

OS  CO 

co  .a 
ex.  a) 


to 
•a  -a 
u 

o  o 

pel  O 
co  o 

*  » 
00  41 
fS  -H 
■rl  > 
U  41 
4J  U 


•h  .c 


_  41 

•9  A 

CO 


u  a  t3 


01  i-l  rH  ^->. 

CO  CO   (0  41 

O  C  -U  O 

Si  i-l  i-l  i-l 

4J  00  O.  > 
•Huh 

C  U   O  41 

m  o  si  m 


41 

e  js 

o  u 
u 

<u  U 

o 

u 

41  • 
41  » 
£  41 

co 

.*  > 

u  41 
O  M 

VI  4J 
41  *rl 

CO  3 

•H  a- 

43 

CO 


Vi 

o 

CO 

a 

4)  4) 


41 
■H 

Vi  O 

U  OS 

41  CO 

VI  04 


3  CO 

o  -a 

I 

u  u 
a  o 

•H 

Vi  Vi 

o.  o 

4-1 
CO  o 
CO 

4)  i-l 
>  rH 

at  o 

si  o 


t-H 
I 


Vl   41  4J 

41   CJ  CJ 

>  a 

41  CM 

41    CO  <4-l 

4-1    CU  0) 
41  Vi 
H   Pi  (J 

a.  co 

a  4i 

O  J3  41 

U 


co 
z 

O 
i-l 
H 

CO 
W 
3 

cr 


H 


H 
O 

s 

< 
z 
o 

I— I 

H 

o 
z 

3 
lb 


CO 

« 

■H 

3 

CO 

(0 

c 

C 

o 

cu 

a) 

■H 

c 

Vl 

4-1 

o 

CN 

c 

« 

1 

o 

•H 

CU 

■< 

o 

O 

o 

O 

o 

o 

oo 

M 

I-l 

00 

c1 

p-l 

i 

•H 

< 

-o 

■o 

a 

o 

c 

u 

o 

CO 

T4 

I-l 

co 
w 
oi 
3 
O 
U 

o 
o 
at 
cu 

J 

o 


z 
o 
o 

§H 

M 
-J 

•U 


TJ 

o  >  c 

«H  H  4 
O 

CO    00  M 

a  4)  «> 

OUT) 

00  o 

a)  41  o 

M  rH 

TJ  CO 

«  h  a 

*j  a  -h 

<d  o  oo 

B  43  tH 

00  CO  M 

■H  O 
CO  CO 

II  tl  tl 

TJ   CJ  43 

c  u 

O  31 

•u   U  C 

4)  41 

h  u  I 

«l  w  3 

TJ  iH  44 

i-i  a  4i 

o  x> 


C3  ■  «• 
rl  ^  «l 
O  rH 
G  43 
l-i  to  -H 
CO"  O.  CO 
g0  4)  CO 
ID   U  O 

4i  u  a 

0  00 

01  tH  W 

a  tj  t> 


u 

41  00 

43  rH 

o  at 

U  3 

3  B 


■H  41 

h 

41  3 

c  -a 

•H  4) 

a  o 

u  o 

41  Vi 

44  Cu 
41 

Q  >M 


c  >. 

O  U 

co  <d 
u  oo 

41  00 
Q.  41 

cj 

00  41 

c  o 

•H 

u  oo 

O  iH 
l-l 

-rl  00 

b  c 

O  T-l 

a  s 
i-i 

4i  <d 

43  U 


TJ  00 

41  41 

41  i-l 

B  44 
r-l 

00  3 

rH  o 

td  i-i 

5  <m 

B  cm 


oo  u 

C  4) 

■H  43 

TJ  U 

0  o 
o 

41 

CO  Vl 

1  09 
< 

a  4i 

cj  u 

rH  41 


00 

B  n 

i-l  o 
l-l 

09  - 

TJ  00 

Cu  B 

3  i-l 

44 

"d  td 

41  TJ 

41  Cu 

B  3 


Vl  TJ 
O  B 
w  o9 

>n  oo 

rH  iH 

a  oo 

O  O 

-O  a 

B  00 

09  09 

>-.  i-l 
TJ  • 

00 

u  u  c 

U  O  O 

td  <m  i-i 

M  l-l 

u  -d  4 

09   4)  CJ 

43  rH  iH 

td  i-l  lw 

3  tH 

41   0,  U 

00  Oi 

I-i  09  > 
09  44 

43   CJ  00 

O   09  O 

0  h  n 

■HUT* 
73   CO  O 
43  O 

r-.  td 

1  4) 

m  41  m 

a  3 

«-l  10  Tt 
U   1  II 


TJ  cq 

SI 

U  09 

41  r-l 

o.  u 

M 

41  41 

43 


•  o 

oo  olJ 

a  co 

1-1  04 

09 

rH  CM 

o  o 


41 
CO 


41 


CJ 
O 
VI 

cu 


c 

3 

31 

Cu 

oo 

00 

4) 

B 

g 

44 

B 

CJ 

O 

o 

u 

iH 

B 

i-l 

M 

4J 

td 

Ss 

41 

44 

td 

m 

IW 

CO 

td 

cu 

s 

u 

1-1 

41 

o 

OS 

a) 

oo 

44 

U 

-rl 

TJ 

43 

43 

t-l 

a. 

CM 

u 

09 

41 

-H 

oo 

O 

44 

(3 

4«! 

Tt 

CM 

)-• 

-rl 

00 

4) 

14 

■H 

4) 

r-l 

to 

44 

O 

» 

CJ 

r-l 

O 

MH 

a> 

•H 

xl 

4) 

> 

U 

43 

B 

r-l 

00 

o 

a) 

41 

41 

o 

r-l 

CO 

m 

u 

CO 

3 

3 

B 

l-l 

4) 

41 

TJ 

CO 

•H 

ft 

iH 

> 

41 

t) 

o. 

O 

CJ 

TH 

M 

to 

43 

O 

09 

O 

0 

u 

09 

U 

rH 

O 

43 

o. 

O 

O 

rS 

1-1 
o 

41 
43 

3 

3 

CM 

44 

OJ 

-H 

«l 

i-i 

i-i 

Tl 

44 

44 

l-l 

> 

> 

> 

o 

td 

o 

4) 

4) 

z 

TJ 

cm 

pcS 

M 

c-4 

ci 

a 

i*i 

>. 
CO 

l-l 

t-i 

44 

as 

4) 

O 

td 

00 

VI 

43 

3 

a 

44 

1 

09 

0J 

a 

B 

UH 

tH 

a 

tH 

rH 

td 

m 

O 

TJ 

3 

> 

«! 

U 

rH 

41  H 

I 

41 

CJ 

4) 

CJ 

1 

r-t  3 

43 

a 

M 

CO 

CO 

rH 

tH  O 

l-l 

E 

OB 

-rl 

4J 

• 

rH 

<4-l  O 

00 

4) 

a 

00  43 

■4H 

B 

a 

B 

44 

•3 

B  O 

4) 

o 

41 

B 

09 

■H  td 

tH 

tH 

tH 

"oo  o 

rH 

tH 

rH 

U  41 

• 

o 

4J 

44 

B  erf 

CJ 

V4 

a 

0) 

09 

1-4  CO 

41 

rH 

O  VI 

tH 

>> 

o. 

CJ 

"0  Ph 

43 

at 

41 

o.  o 

04 

IH 

tH 

B 

u 

o 

4-1 

O.  M-J 

r-l 

09 

r-l 

CM 

41  M 

09 

09 

3 

CJ 

iH 

td 

tH 

CU  4t 

>w 

tH 

O 

00  TJ 

TJ 

u 

44 

43 

O 

<4H 

iH 

41 

rH 

41 

CD 

14 

rH  44 

rH 

44  S 

09 

a  tj 

41 

rH  O 

B 

TJ 

a. 

at  tH 

tH 

IH 

CV 

CJ 

tH 

O 

B 

3 

43  at 

41 

31 

fc, 

P3  VI 

•H 

td 

TJ 

14  44 

TJ 

44 

O 

:  o 

4-1 

B 

41 

B 

VI 

06 

id 

rH 

4-1 

rH  tH 

Be 

tH 

O 

co 

oo  - 

44 

• 

td 

C9 

09  td 

CM 

CU 

-  oo 

O 

T» 

44 

43 

44  1 

09 

rH 

tH  14 

B 

4) 

iH 

14 

iH 

09 

4 

O  41 

■H 

a. 

a,  og 

<4H 

CJ 

1 

44  41 

VI 

B 

09 

44 

00  iH 

O 

CO 

tH 

44 

td  43 

o 

41 

o 

a 

O 

tH 

cd 

3 

B  oo 

4H 

•O 

43 

41 

43  a 

a 

■4H 

rH 

0 

tH  M 

•0. 

00 

0 

o 

O 

43 

T3  tH 

c 

IH 

43 

at  • 

43  tH 

iH 

43 

44 

tH  O 

tH 

O 

44 

TJ 

44  44 

44 

44 

44 

tH 

O  3 

•C3 

tH 

rH  OJ 

tH  0) 

09 

09 

3 

O 

Vi 

T3 

3 

09  tH 

3  44 

CJ 

43 

CJ  VI 

O 

41 

CJ  <4H 

B 

•tH 

44 

TJ 

41 

o 

B 

>> 

00  tH 

-**  9! 

CM 

>s 

tH 

4tS  43 

CJ 

00 

CM 

tH  44 

il  1 

■H 

•M 

44 

09 

CJ  ~» 

tH 

•H 

UH  B 

44 

Tt 

B 

Pa 

4)  00 

14 

CO 

VI 

41 

VI  CJ 

tH 

IH 

0J 

43  tH 

o 

00 

41 

U  TJ 

4)  O 

41 

41 

60 

41 

O  43 

U-l 

td 

> 

O  iH 

>  TJ 

CJ 

> 

09 

43 

00 

B 

0J 

0 

44 

|4 

-rl 

o 

B 

3 

M 

td 

41 

OS 

TJ 

td 

4) 

t4 

to 

O 

3) 

o 

CO 

o 

ERRi 

CJ 

TH 

M 

>. 

at  oi 

o 

CM 

O 

T) 

td 

44 

a 

a.  co 

14 

Tt 

CM 

31 

TJ 

31 

VI 

04 

CV 

4J 

3 

31 

o 

OF 

M 

tH 

41 

>M 

43 

•M 

3  v. 

B 

41 

O 

tH 

O 

CO 

•H  OJ 

41 

O 

4-> 

SO 

<d  Cu 

ICES 

4-1 

td 

VI 

u 

B 

rH  O 

44 

00 

B 

14 

41 

o 

■H 

O  IH 

tH 

B 

•H 

CM 

3 

rH 

a 

VI 

tH 

TJ 

44 

00 

rH 

>> 

3 

rH 

U 

B 

B 

S 

•rl 

43   44  • 

§ 

rH 

O 

31 

cd 

o 

43 

3  B 

CO 

44 

-rl 

o 

tH 

M 

Vi 

0J   o  O 

B 

43 

a 

4-1 

4-1 

cm 

O 

TJ  43  tH 

41 

td 

44 

09    44  44 

tH 

01 

44 

TJ 

B  iH  td 

CJ 

VI 

3 

O 

31 

44 

3  o 

M 
H 

•H 

3 

•H 

rH 

3) 

■H 

CJ 

44  tH 

U4 

00 

> 

td 

Vi 

CM 

to 

B    rS  CM 

POTEl 

CM 

00 

4J 

Vi 

!-i 

•H 

Vi 

41    CJ  TH 

3 

e» 

W 

31 

O 

U 

44 

I    B  44 

to 

TJ 

CJ 

U 

CO 

rS    41  VI 

B 

O 

>» 

cu 

B 

3) 

43 

09   00  41 

rH 

4J 

43 

r-l 

CJ 

CO 

04    09  CJ 

TJ 
4) 

§44 
rH  B 
rH  41 

O  a 

CM  44 

u 

41  09 
Vi  Cu 
CO  41 
TJ 

CO 

31  00 
Vi  B 
3  -H 
TJ  rH 
41  rH 
CJ  tH 
O  43 
Vl 

Cu  41 
43 


43 


rS  U 

43  O 


C  at 
O  B 


td  td 

rH  CJ 
CJ  iH 


31  Vl 
B  31 
TH  CJ 


TJ 
0J  (3 

Vl   09  O 

3  OS 

oo  3  co 

B     31  04 

41  iH 

>  41 

O   41  43 

44    |4  44 


B 

3) 

TJ 

•H 

> 

41 

td 

at 

3 

4J 

43 

o 

M 

B 

rH 

O 

-rl 

B 

3) 

rH 

l-l 

O 

00 

O 

td 

-H 

to 

% 

CM 

c 

U 

CJ 

CO 

-H 

u 

td 

0 

4-1 

31 

TJ 

1 

o 

44 

3) 

u 

td 

IH 

IH 

-rl 

4H 

6  43 

o 

TJ 

Ctt 

OO 

la 

01 

B 

me 

4-1 

CM 

41 

CO 

CJ 

CJ 

tH 

3 

B 

4-1 

43 

31 

TJ 

CJ 

•H 

3 

44 

VI 

3 

oo 

VI 

O 

Cu 

3 

41 

B 

o 

TJ 

rS 

B 

o 

TJ 

■H 

iH 

o 

rH 

tH 

u 

41 

>. 

o 

00 

M 

CJ 

CM 

B 

CO 

41 

44 

o 

U 

•H 

31 

-rl 

rH 

43 

B 

M 

4J 

43 

44 

3 

4J 

31 

Cu 

41 

U 

44 

t4 

CJ 

00 

43 

31 

o 

iH 

41 

44 

4-1 

CJ 

00 

Cu 

44 

U 

B 

cd 

31 

Cu 

VI 

31 

3) 

>N 

c 

O 

3 

td 

43 

41 

43 

Q 

00 

a 

3  43 

31 

<0 

s 

C~  14 

•rl 

rS  3 

4) 

4J 

TJ 

id  to 

VI 

Vl 

44  00 

ca 

31 

Vi 

o 

00  09 

43 

o 

o 

u 

1 

4-1 

CM 

CJ 

o 

CM  O 

II 

CM 

O  44 

-rl 

O 

>> 

> 

1 

cd 

4J 

TJ 

31 

« 

43  TJ 

rH 

cd 

•H 

CJ 

4J  O 

o 

14 

41 

u 

00  43 

o 

Vi 

C— 

3 

S  44 

at 

e— 

tH 

B 

VI 

O 

4)  41 

44 

TJ 

M 

CO 

O 

to 

rH  a 

cd 

41 

O.  tH 

-rl 

4) 

CJ 

44 

4-1 

43 

« 

00  01 

-rl 

4J 

Vl 

a 

cd 

44 

4J 

a 

rH 

tH 

o 

TH 

CJ 

td 

-rl  41 

Cu 

a 

4-1 

cd 

•H 

CO 

TJ 

rN  Vl 

3 

43 

CJ 

rH 

CM 

iH 

cm  41 

T) 

3 

31 

y 

tH 

a 

iH  43 

00 

rH 

14 

44 

44  44 

44 

rH 

41 

M 

at 

M 

M 

a) 

44 

O 

43 

4) 

O 

09  00 

43 

o 

CJ 

44 

CJ 

§ 

44 

CJ  M 

44 

a 

00 

td 

at  43 

Vl 

3  3 
00  4) 

go  a  t-i 

CO  -rl  > 
td  41 

O  rH  14 
44  CJ 

O 

>.  >,  Oi 
09  Vi  CO 
3   41  04 

09   41  - 
TJ 

31  |4  4) 
M   O  tH 

CC)  cu  144 

43  -H 

4-1    44  44 

C9  tH 

CO  43  41 

M    44  CJ 


Vi 

td 

-H 
TJ 
41 

a 

VI 
31 


B 

41  B 
•H  4) 
44  0J 
41    Cd  43 
43    O,  c— 
14  O 


a) 

aj  Vi 
Cu  31 
TJ 
14  41 


B 
41 
00  14 

09  O 


O  oi 
B  CO 
04 

41 

>  4) 
at  43 


tu  43  44 


43  Ss 
CJ  43 


B  09 

41  « 

41  O 

43  a 


rH  43  TJ 

CO  rH  3  41 

CJ  CO  tH 

CO  44  09  CM 

tt  <t  iii 

CM  O.  -rl  44 

00  01  VI 

Vl  O  rH  41 

O  43  CJ  CJ 


^ — s 

TJ 

oo 

31 

CM 

■H 

TJ 

3 

O 

>■ 

00 

at 

c 

C 

H 

o 

31 

CO 

•H 

s 

VH 

s 

Vi 

4-1 

o 

> 

ctt 

3 

CH 

B 

■H 

rH 

CO 

TJ 

1 

O 

44 

H 

iH 

3) 

< 

CJ 

at 

a 

CJ 

a 

CJ 

rH 

CJ 

O 

o 

•H 

rH 

< 

0£ 

M 

rH 

00 

CM 

T-l 

B 

04 

1 

1 

iH 

CO 

iH 

3B 

< 

44 

TJ 

T) 

a 

tH 

3) 

O 

B 

B 

o 

41 

43 

O 

tfl 

-H 

rH 

U 

4-1 

< 

s 

y — • 

z 
o 

-* 

in 

rH 

rH 

rH 

H 

w 

-  - 

O 

Z 

3 

fc. 

41 

41  43 
43  44 
00  44 
4)   Ss  O 
43  at  B  44 

4-1  TJ  O 

00 

00  cm  TJ  B 

B   O  41  -H 

iH  rl  O 

TJ   U  cm  00 

Vl   «  iH 

O  43  44  rH 

CJ    B  Vl  rH 

41    3  41  lH 

04   B  <J  43 


>M 


00 
B 
•rl 

TJ  VI 
tH  O 
O  Ck 

o 

4) 

Vi  • 
B 

TJ  O 
B  tH 
09  44 
CJ 
00  B 

B  3 

Cu  tH   CO   4J  -rl  cm 


o 
a) 

VI 

41  44  44 

>  VI  CO 

tH  CO  43 

44  O.  09 
09 

t-H  31  td 

41  44  44 

Vl  09  09 
Vl  TJ 
09  09 

Cu  41 
CO  41  43 
CO 


M  td 

41        rH  Vl 

>       rH  4) 

4)  44    09  Tt 

Sr.    3  00  41 

u   O  3   44  Cm 

■rl  43  8  09 

rH             43  Vl 

3-  O    44  O 

rH  Oi  <4H 

CJ"  O  CO  rS 

t4  04  CM  CO 

09   44         tH  rH 

44    B  4)    VI  rH 

09   O  43   41  iH 

TJ   CJ  44   >  43 


O 
td 

V4 


TJ 
41 
tH 

cm  tH 
•HOT) 

44    3  CM  CM  3) 
4)  Vl   41  O 

tH  41  iH  Vl  CTJ  4J 

C9   CJ   >   O   VI  U  00 

31  M   31  tH  43 

CO        U   cd  43  TJ  CO 

44  rH          B    §  B 

B    VI    41  -rl    3  -rt  ft) 

41   4)  43  TJ   B  43 

rl  Q.  41  u  00  44 
VI  O        O  41 

09  tH   rV  O  43  09  B 

Cu  Cu  43   CJ  M  TJ  O 


cd 

y 
a 

cu 

he 

4-J 

c 

4-1 

00 

Irt 
01 

1 

*■ 

CO 

or 

ch 

I 

-H 

4J 

X 

0) 

o 

13 

J3 

■rl 

CU  CU 

■H 

e 

4J 

a 

CO 

cd 

1-4 

rH 

M— ) 

0) 

4J 

X 

rH  CO 

0) 

-s 

•A 

cd 

pa 

ch 

cd 

X 

u 

4-1 

T3 

a 

CU 

he 

O 

43 

4-1 

(3 

cd 

01 

4-1 

*H 

M 
o 

iH  3 
4-4 

4J 
O 

3 

> 

CU 

t4 

& 

e 

00 

CO 

01 

4J 

M 

cd 

4-1 

o 

CO 

CO 

td 

V-i 

<T 

cu 

ta 

43 

01 

cd 

Xi 

4J 

oo 

3 

as 

•a 

CJ 

CJ 

cu 

00  iH 

rJ 

o 

rH 

4-1 

3 

cd 

00 

cd 

O 

43 

a 

4-> 

-r-l 

• 

rH 

d 

4J 

cu 

I 

ao 

d 

> 

tj 

4J 

rH 

Tj 

4-1 

id 

cd 

a 

X 

3 

T-i 

-iH  CU 

G3 

cu 

rC 

M 

JRES 

>> 

■H 

a 

cu 

cu 

3 

01 

3 

01 

d 

D 

CD 

U-4 

T3  rH 

•3 

4-1 

4J 

rH 

09 

c0 

o 

1-4 

4-1 

43 

o 

<! 

4J 

rd 

a 

0) 

■rl 

»H 

CO 

d 

d  *h 

cd 

4J 

01 

4J 

0) 

(U 

4J 

4-1 

00 

> 

c 

CU 

CU  M-4 

X 

•>» 

00 

d 

CD 

a 

rH 

0) 

rH 

jj 

0) 

cu 

cd 

03 

01 

01 

o 

4-1 

rO 

o 

o 

4-1 

o 

& 

c 

U 

• 

u 

4-1 

rH 

PL. 

•rl 

C 

u 

to 

cd 

u 

* 

0> 

)CEI 

rH 

CJ 

cd 

i-i 

a 

■H 

cd 

13 

a 

•H 

cd 

&' 

J3 

•rl 

■M 

t3 

rH  CU 

ao 

-G 

d 

tj 

ca 

o 

u 

cu 

o 

s 

4-1 

a 

01 

4-1 

rH 

Xi 

rH  rH 

0) 

4-4 

•H 

a) 

CO 

o 

1-1 

u 

CJ 

■JQ 

co 

o 

J3 

0> 

al 

cu 

J-* 

-H  r^i 

■H 

CO 

W 

a! 

rH 

cu 

■H 

XI 

<U 

CJ. 

4-1 

I— 1 

a 

J>l 

o 

rQ  CJ 

4-1 

!-l 

00 

a) 

i— 1 

43 

a 

CO 

a 

d 

H 

cu 

CQ 

T3 

a 

rH 

■rl 

i-i 

CO 

•H 

■rl 

CU 

cd 

0-. 

ij 

cd 

4J 

CO 

cd 

■rl 

Cu  43 

rH 

i-4 

0> 

■rl 

4-1 

o 

t3 

M 

> 

X 

ITROL 

a) 

M 

« 

3 

FJ 

"tj) 

3 

CJ 

9 

U 

CU 

CO 

•rl 

a 

*0 

ao 

O 

TJ 

a 

o 

C 

c 

J3 

M 

O 

rH 

cu  cu 

4-1 

0 

CU 

■-I 

co 

c 

1-1 

4-1 

a 

CJ 

H 

td 

43 

•H 

■rl 

>> 

4-1 

01 

O 

rM 

l! 

O 

?» 

o 

TJ 

cu 

CU 

0) 

CJ 

3 

3 

oo 

9 

ao 

rH 

-r-l 

> 

0) 

o 

cd 

XI 

UJ 

M 

o 

00 

•9 

4-J 

cd 

• 

rH 

01 

O 

GO 

el 

a 

3 

o 

M 

■H 

4-1 

cy 

•H 

s~ 
O 

4-1 

s 

cd 

r4 

CI! 

b 

,4=1 

43 

u 

4-1 

a) 

00 

o 

4-1 

>.  rH 

> 

C 

4J 

CO 

-H 

JJ 

CJ 

CJ 

ca 

o 

u 

n 

s 

0) 

v—1 

00  » 

rH 

i-l 

cj 

O 

0) 

GO 

4-1 

00 

■rl 

o 

u 

o 

cd 

.'j 

a 

4-1 

td 

0 

CU 

d  CO 

•rl 

CM 

CJ 

a 

• 

o 

•H 

XI 

l*-t 

0) 

4J 

4-1 

Cu 

^3 

XI 

o 

a! 

ct! 

o 

CO 

T3 

*H  CU 

Cd 

a 

t— 1 

B 

cd 

«H 

4-1 

M 

s 

<u 

O 

ss 

i — j 

CJ 

-H 

4-J 

*0  rH 

X 

0) 

CU 

>> 

« 

o 

•H 

O 

CO 

4-1 

O 

-r-l 

!u 

iH 

(3 

rQ 

■H 

t3 

CO 

rH 

d  -H 

03 

Ij 

rQ 

M 

u 

4-4 

■M 

■rl 

0) 

TJ 

0) 

cu  -a 

-C 

■H 

o 

X) 

cy 

>s 

O 

CU  4-J 

1-4. 

i 

CD 

H 

r4 

43 

o 

0) 

rJ 

oo 

• 

u 

c 

t3 

CO 

c 

CU 

c 

O 

< 

> 

a 

>> 

in 

CO 

O 

4-1 

as 

IH 

o 

0) 

X 

CU  iH 

-H 

rH 

T-i 

CD 

4-1 

i-t 

•<-< 

CO 

0) 

2 

o 

4J 

00 

o 

CO 

0 

CO 

G3 

cu 

u 

cu 

CO  cd 

a 

4-1 

cy 

4-4 

M 

4J 

1 1 

cd 

CU 

P-i 

•H 

4-1 

CO 

o 

EH 

1-5 

Cu 

60 

01 

o 

o 

rH 

kS  °H  OJ 

a 

0) 

Q> 

O 

a 

cu 

CO 

CO 

MH 

C 

4-1 

4-J 

CJ 

1-1 

CJ 

CO 

J-i 

rH 

oo 

•H 

co  a* 

eg 

CU 

V-I 

< 

i-H 

>■ 

43 

•H 

■H 

co- 

CD 

CD 

cd 

CO 

rH 

CO 

rH 

td 

XI 

rH 

to 

El. 

O    O*  rH 

JS 

cO 

o 

-H 

CO 

cd 

4J 

X 

rd 

Xi 

i — ( 

i-i 

rH 

1-1 

Xi 

1-1 

cd 

M 

d  cd 

Cl 

oo 

3 

UH 

c 

5-j 

4-1 

a, 

4-1 

r-l 

cd 

O 

4J 

0) 

0 

S! 

O 

T3 

CO 

00 

>* 

a) 

4-4 

00 

01 

cu 

O 

Q) 

a 

CO 

cd 

at 

CD 

a 

Qj 

oo 

•H 

Cd  T3 

M 

u 

CU 

X 

o 

43 

X 

o 

43 

o 

o 

x> 

c 

>.  CO 

Xi 

■rl 

o 

cd 

cd 

rC 

■h  d 

O 

o 

rd 

> 

CU 

M 

-1 

H 

•rl 

o 

4-1 

4-1 

CJ 

cd 

cd 

43 

PU 

cd 

4-1 

X 

cu 

1 

a 

CJ 

H 

>U 

3 

4J 

0) 

Xi 

CO 

PS 

H 

o 

H 

CO 

rH 
r4 

til 
O 

a 
X 

o 

►J 
o 


55 
O 
U 


o- 
o 

OS 

CO 

PU 


>1 


H 
O 
< 

< 
ta 

< 

rJ 
< 

2 
O 

M 

H 
U 
Z 
=> 


1 

3 

CD 

cd 

CU 

4-1 

4-1 

1-1 

OS 

CJ 

e 

CO  > 

o 

cd 

cu 

4-J  CU 

u 

a  u 

c  u 

4-1 

3  CU 

cu 

CO 

CJ  U-4 

•H  4-1 

XI 

O  CO 

4-1  C 

cd 

X  (3 

cd  cu 

O 

m 

CU  iH 

00  M 

O 

c/1 

ca 

C  4-1 

14H  -rl 

W 

4-1 

•H 

O  MH 

CO 

U 

01 

4J  cd 

CH 

rH 

cu 

U  4-1 

3  3 

O 

SOU: 

Xi 

O  cd 

cu  co 

u 

CO 

eu-o 

•H  a 

4-1 

r*i 

Cu 

>  -H 

c 

H 

3  00 

CU 

o 

[AL 

O 

CO  c 

fU  <4U 

o 

3 

•H 

O 

V-i  M 

>> 

H 

MH 

0)  3 

rH  CU 

4-1 

Z 
td 
H 

O 

Xi  XI 

CU  CD 

1-1 

4-1 

a  » 

> 

CO 

o  c 

•H  CO 

■H 

00 

o 

4-1  CJ 

4-1 

O 
CU 

o 

M  iH 

Cl  CU 

CJ 

r4 

O  4-1 

»  43 

cd 

CO 

0) 

o- 

r-l 

4-1 

TJ 

4= 

00 

CO 

0) 

01 

c 

4-1 

C  MH 

a 

M 

C 

O) 

cd 

o  o 

1 

0 

•H 

o 

a 

CO 

Xi 

0) 

o 

•H 

r-l 

T3 

4-1 

o 

V4 

4-1 

o 

Xi  cu 

cn 

u 

l 

u 

4-J 

4J  a 

V-i 

rH 

01 

MH 

O 

CO 

CO 

4-1  lu 

rW 

3 

01 

cu 

CO  r4 

o 

cd 

•H 

1-4 

4-1 

a 

V4 

X 

iH  3 

r4 

00 

u 

4-1 

4-1  O 

3 

C4U 

01 

rH 

•H 

•H 

01 

01 

3  XI 

o 

CD 

a 

13 

C  CH 

c 

1-1 

X 

-H 

c 

XI 

Cu 

4-1 

CU 

3 

cd 

CU 

% 

CU 

o 

CU 

1-1 

•H 

O 

CO 

o 

M 

cd 

00  CJ 

u 

cn 

Cu 

a  cu 

o 

01 

o 

M 

a 

o 

co 

e  o 

0) 

o 

cd 

3 

3  -C 

X 

CU 

4-J 

o 

X) 

a 

O  M 

43 

4-1 

0) 

CO 

CJ  4-1 

CO 

01 

CO 

c 

>. 

o 

rJ 

■rl 

rH  CU 

4-1 

co 

rH 

M 

01 

5= 

0 

cu 

c 

■H 

4-1 

4-1 

o 

O 

rH 

cd 

CO 

u 

0) 

cti 

CJ 

O 

•o  c 

4-1 

tJ 

CO 

1-1 

iH 

o 

CU 

TJ 

00 

4-1 

a 

0 

cd 

t-4 

iH 

a 

cd 

01 

X 

rH 

3 

01 

X 

>>  cu 

c 

CJ 

<a 

■H 

CD 

r-l 

H 

00 

c 

4J 

M 

a 

3 

cd 

CU 

r4 

4-1  C 

iH 

cd 

i-i 

4-1 

4-J 

CU 

CO 

c  cu 

C 

O 

iH 

cd 

3 

•H 

r>» 

•H  iH 

rH 

u 

4-1 

o 

w 

•H  43 

u 

•H 

MH 

4-1 

a* 

> 

rH 

H 

rH  14H 

rH 

4-1 

cd 

cd 

CO 

01 

c— 

4-1 

o 

C 

CU 

cd 

M 

•H  CU 

O 

CO 

Cu 

IJ 

4J 

C 

o- 

rJ  r». 

4-1 

co 

U-4 

cd 

cd 

u 

CJ 

cd 

XI  X) 

M 

rQ 

13 

J-J 

cd 

o 

o  cd 

cd 

rH 

o 

4-1 

•H 

01 

•H 

U 

CO 

rH 

1-t 

CD 

i-i 

iH 

&  a 

c 

cd 

0) 

cd 

Xi 

X 

r-l 

CO 

C 

r-l 

Xi 

u 

4J 

cu 

1-1 

CO 

4-J 

X 

X 

-a 

4-J 

CU 

o 

C  rH 

O 

u 

cd 

X 

cd 

Cu 

4) 

3  Xi 

X 

1-1 

1-1 

o 

iH 

o 

a 

O  M 

CJ 

o 

0) 

o 

rH 

CO  CJ 

u 

0) 

CU 

•H 

f> 

o 

XI 

0) 

Cu  cd 

4J 

3 

4-1 

u 

iH 

o 

0) 

CO 

U 

O 

ai 

X 

> 

CO  CU 

u 

X 

cd 

01 

cd 

CU 

CO  xi 

o 

J3 

o 

01 

u 

CO 

o 

c 

cd 

0J  rH 

o 

a 

43 

1-1 

Xi 

a 

o 

HI  3 

CJ 

en 

Xi 

CU 

CU  Cu 

a 

cd 

U  CJ 

<4-l 

cd 

4-1 

> 

4-J 

cd 

CJ 

CO 

ao 

Oil 

»J 

C 

C 

CO 

iH 

■H 

Q 

lu 

3 

X 

X 

O 

C 

a 

ct) 

S 

cu, 

cn 

3 

a 

o 

cn 

(SI 

O 

fn 

01 

•H 

0 

>4H 

4J 

G 

o 

O 

CO 

o 

1-1 

4-1 

•H 

PU 

rH 

c 

4-1 

O 

01 

CJ 

3 

u 

0 

CU 

01 

4-1 

3 

rH 

iH 

c 

O 

rH 

> 

o 

o 

o 

01 

o 

a 

CJ 

PS 

CO 

w 
05 
z> 
a 
w 
o 
o 

05 
PL, 

o 

05 
H 

z 
o 

CJ 

>< 

H 
(-1 

< 


0   01  K 

>>  00  a. 

CO  M 

cd  cu 

CO  1— I  > 

■H  i-l 

JG    Cd  4J 

4J  O 

JG  CO 
M-i  4-1 

O  *H  rl 

3  rl 

G  cd 

O  >-i 

i-l   O  J2 

4J    4-1  4-1 

cd  co  1-1 

1-1  G  3 
H  *H 

cd  Tl  01 

>   M  rl 

O  i-l 

<   O  y-l 
CJ 

H 

•   01  01 

u  si  xi 

O    4J  U 

4J  O 

CO  en 

G     01  W 

•H  Tl  O 

TJ  -H 

U  >  ^ 
OOO 
QUO 
CJ  O. 


cd  cd 

»-l  .G 

H  U  CJ 

a  o  a 

CO  01 

o 

a  c  co 

cd  4J 


CO 


3 
01 

iH  > 
<4-l  > 

o   01  01 

w  Xi 

00  cjo 
c  g 

i-l  01  01 

3  Si  Si 

4J 

H  CO 

o  a 

t-i  §  01 

O  CO  • 

0  U  3  G 
i-l  o  o  o 

1H    41  J  tl 

CO  I  4J 

m  c  c  cd 

01  i-H  *H  4J 
TJ  *d  CO 

C  m  co 

3   O  4J  CO 

o  c  - 

TJ    o  01  01 

0)  i-l  CO 

rl    U  44  VI 

t4  si  cd  3 

w  H  B,  C 


r|  Cd 

rl  G 

■H  i-l 

X)  M-t 

TJ  rl 

G  -rl 

Cd  4J 

a 

CO  3 
•H 

CO  TJ 

o  01 

a  a 

00  t-i 

cd  cd 


cd 
o 

•h  c 

>M  01 

i-l  0» 

4J  ^5 

M 

01  CO 

0  cd 
si 

u 

O  4-1 

M-l  C 

01 

•a  ih 

G  4J 

cd  cd 

o. 

4-4 

u  01 

cd  si 

H  4J 


TJ    C  4J 


a  a 
01 

4J  0) 

CO  H 
>,  cd 
(0 

CO 

Vj  01 
01  rl 
JG  *H 


CO  01 
JO  0 
cd  G 
o 

01 

Si  rl 


M-l  43 
O 

01 

c  x: 

O  4J 


01  00 

a 

M  i-l 

01  T> 

Tl  G 

a  01 

3  CM 


01  O 


a,  a 

e  o 

O  -H 
CJ  4J 


SNT) 

a.  01 
o  g 

CJ  -H 

cd 

00  4J 

g  a 

■2  3 

o 

3  co 

CO 
-  4-1 

w  u 
o  m 

4J  w 

CO  4J 

c  co 

i-l  J3 

-o  cd 
u 

O  t-l 

o  cd 

CJ  3 
4J 
0)  cj 
Si  CO 
H 

M 
O 

■a  - 

0>  4J 

00  01 
)-i  01 
CO  Si 

si  co 
U  .M 

CO  )-i 
i-l  o 
tj  3 


4J 

si  01 
ao  si 


t 


si 

a  o 

01  -H 
4J  JG 

co  3 
>. 

CO  c 
00 

a  <d 

i-l  rl   0)   00  CJ 


cd  CO 

cj  x> 

tH  cd 

TJ 

01  a 

0  cd 


a) 


r|  Cd 
4-4 
-rl 
Pi 

CO 

O 
si  si 
o 

3  01 

CO  CO 

o 
si 
•  u 
m 

01  c 


CM  rl 

3 

01  M-l 

CO  p. 

01  rl 

si  o> 

4->  JG 

C  01 

1-1  ja 


jg  a  o> 

4J   -H  rl 

CO  rl 

a  s  o 

iH  CJ 
CO 

4J  rl  CO 

O.   CO  4J 

0)     4-1  CO 

-H  TJ 
D. 

4-4    CO  O 

O   O  05 

C  £  01 
0-, 

CO  01 

i-4   CO  U 

o  o 

si  m-i 


01 
01 

jg  e 

CO  1-4 

J<i 


> 
u 

d  01 
cd  co 


VI 

01 

I-t 
M 

CJ 
iH 

01  4J 

a 

O  T3 
CO  c 
cd 

01 

>  CO 

cd  00 
Si  o 

rl 

CO 

T3  4J 
V4  CJ 

cd  cd 
cj  u 
4-1 

rl  CO 

O  Xi 

u  cd 


c 
o 

CJ 

J4« 
14 

o 
3 

3 
ot 

•H 
> 
CD 


5 


^  01  I 

m  J3 
H  >4-i 


5) 
> 
O 

09 
0) 
00 
oj 

4-1 

c 

CO 
> 
T3 
05  cd 


CO 
•  tH 

-a  x 

01  01 

00  ; 
cd  a 
a  01 

CO  4J 

a  a  > 
>> 

CO 

rl 

-H  Cd  • 
CO 

cd  si 

01  CJ  ■ 
3 

0)  CO 
M 

cd  14-1  1 
o 

>. 

01  01 
Si  rl 
4J    O.  ' 

a 

4J  CO 

cd  H  ■ 
xi  01 

4J 

!>»  • 

G  m  ; 

-H  CO 
rl 

CO  o. . 

01  a 

rl  01 
*H  X 

CM    O)  ' 


T-l    4-4   i-l  CN| 


CO 
o  i-l 
X 

01 

CO  01 

3  si 
u 

01 

j3  a 

4J  O 
M 

00  M-l 

G 

i-l  -o 
-o  0) 

U  4J 

CO  Q. 

00  M 

01  01 


•  c 
01  01 

00  T3 
CO  *H 

4J 

00  a 

G  01 
•H 

o 


M 

o 
o 

43 

-a 
G 

co  . 

33  rl  O. 

rl 
U    O  CO 
O  44  HI 
4J  4-1 

cd  01  01 

C  JG  rl 


a>  co 

4J  T-l 
CO  H 
rl  CO 

a,  cj 

00  01 

G  co 

■H  01 

CO  Xi 

CO  4-1 
O 

CO 


I  01 

H  JG 

O  4J 

o 


3 

01  G 

•H  O 
> 

01  T> 

H  1-4 

O 

01  0 


01  01 


■i 


o 

01 
cd 

•H  C 
w  01 
01  01 
4J  J3 

cd 

S  co 
CO 
_  JG 

01  TJ 

H  H 

CO  CO 

CJ  CJ 


■a  a 

H  G  O 

OOO 

o 

CJ  XI  01 

01  CJ 

3  T)  -H 

01    3  14-1 


01  CJ 
01 

G  rl 
CO  rl 

o 

co  0 
01 

CO  H 

3  O 
4J 
w  cd 

o  c 

iH 
T3 

G  H 

O  O 
O 


H  iH 

IE 

o 


CO 

G 
O 

i-l  M 

CO  01 


CD  01  iH 

-  M  4-1 

■H  CO  4J 

a  *h 

Cuts  3 

•H  C  T3 

co  cd  cd 


3 

01  c 
G  01 
JG 

U  4J 

o 

>4H  I-l 

o 

4-1 

rl  CO 

00  01  i-l  6 
G  -G  CO  iH 
rl  H  TJ  TJ 


4-1  O 

%  3 

I-l  01 

O  i-l 

CH  > 

c  01 

1-1  w 


a) 

4J  CJ 

01  i-l 

01  -o 

jG  5; 

co  a 

M 

u  01 

0  Xi 

3  4J 

3  C 

01  -H 


>  4J  i-i 

01  a 

M  O)  01 

JaS  t-i 

01  01 

Si  CO  JG 

u  1-1  3 


O  4J 

4J  Ol 

G  01 

O  Xi 
CO 

■0 

V4  M 

cd  O  JG 

o  3 


01  a 

-  o 
o 


cn 

rl 

I 


Q5 

o 

05 


o 

CO 

w 

CJ 
05 
GJ 

o 

CO 
rl 

H 

z 

Ed 
H 

o 


CO 

z 
o 

rl 

H 

CO 

w 


E-" 

CJ 

< 
< 

05 

< 
< 


co 

cm 

oc 

4-1 

c 

c 

CO 

1-1 

1-1 

Q 

1-4 

4-1 

3 

H 

T3 

-O 

0 

G 

p. 

co 

a 

a 

u 

(0 

3 

C 

0 

CO 

CO 

O 

rl 

01 

-H 

0 

<4-4 

4-J 

c 

0 

TJ 

O 

CO 

0 

Vj 

0) 

4-1 

■H 

Cl, 

3 

rl 

c 

4J 

G 

O 

0) 

CJ 

3 

■H 

H 

a 

cu 

01 

4J 

4-1 

3 

rl 

i-l 

G 

G 

CJ 

H 

> 

0 

0 

0 

O 

cu 

0 

CJ 

Q 

O 

m 

[X4 


Instructions  for  Use  of  Work  Control  Card  — ' 


(Exhibits  II-5  and  II-6) 


Current  Review  Section 

Behind  the  "Current  Review"  guide  are  five  additional  guides  labeled  for 
each  day  of  the  work  week.    The  Review  Work  Control  cards  on  the  patients 
whose  current  periods  of  hospitalization  are  being  reviewed  will  be  filed 
in  the  Current  Review  section  behind  the  guide  for  the  day  of  the  week  on 
which  the  next  review  of  the  patient  record  is  due.     The  cards  are  to  be 
filed  by  room  number  according  to  the  nurses'  station  at  which  the  patient 
record  will  be  filed. 

At  the  beginning  of  each  work  day  the  cards  behind  the  guide  for  that  day 
must  be  pulled  and  the  guide  for  that  work  day  moved  to  the  back  of  the 
section  (see  below  for  instructions  on  rotating  guides  for  days  of  the  wee 
Prior  to  going  to  the  nurses'  stations  to  begin  review,  work  is  put  into 
order  by  room  number  according  to  nurses'  station  indicated  on  the  Review 
Work  Control  Cards.    At  the  end  of  the  day,  the  cards  are  filed  behind  the 
guide  for  the  day  of  the  week  on  which  the  next  review  is  due. 

The  guides  for  the  work  days  should  be  rotated  to  insure  that  the  cards 
for  the  work  day  are  in  the  front  of  the  box  on  that  day.    For  example, 
on  Monday  as  soon  as  you  have  pulled  the  cards  move  the  guide  labeled 
"Monday"  behind  the  cards  filed  after  the  guide  labeled  "Friday."  Doing 
this  will  assist  you  in  scheduling  reviews  by  making  it  easier  for  you  at 
the  end  of  the  day  to  file  the  cards  behind  the  guide  for  the  day  that  the 
next  review  is  due.    The  day  of  the  next  review  may  be  one  or  two  working 
days  in  advance  of  today's  date,  depending  on  the  situation.     If  you 
rotate  the  guides  for  the  days  of  the  week,  at  the  end  of  the  day  you 
always  will  file  behind  the  first  guide  those  cards  on  patient  records 
that  must  be  reviewed  on  the  second  working  day  after  that  day's  review. 

Appeals  Section 

When  the  attending  physician  requests  a  hearing  by  the  Local  Professional 
Review  Committee  or  any  concerned  party  appeals  the  decision  of  the  Local 
Professional  Review  Committee,  the  Review  Work  Control  Care  on  the  patient 
involved  will  be  filed  in  the  section  labeled  "Appeals."    Cards  in  this 
section  will  be  filed  in  alphabetical  order  according  to  the  patient's 
last  name. 

When  the  hearing  has  been  completed  and  a  decision  rendered,  note  the 
disposition  in  the  comments  section  of  the  care  and  file  it  in  the 
appropriate  section  of  the  card  file. 
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PLEASE  NOTE;     Cards  on  those  patient  records  that  are  being  reviewed  by 
the  Review  Physician  are  not  filed  in  this  section.     They  are  filed  in 
the  "Current"  section  behind  the  day  of  the  week  that  the  next  review  is 
due. 

Discharges  Section 

After  a  patient,  whose  record  has  been  reviewed,  has  been  discharged  and 
you  have  completed  all  forms  and  reports  regarding  his  record,  file  the 
Review  Work  Control  Card  in  the  section  labeled  "Discharges."    The  cards 
will  be  filed  in  alphabetical  order  according  to  the  patient's  last  name. 
Cards  on  discharges  are  temporary  records. 

NOTE :    Additional  items  recommended  would  be  to  include  the  date  abstract 
was  completed  and  the  date  bill  was  certified.    A  disadvantage  in  this 
system  would  be  the  possible  loss  of  a  card,  at  which  point  a  retrospective 
review  would  have  to  be  conducted.    However,  the  review  coordinator  could 
check  numbers  of  cards  in  current  review  against  hospital  census  lists  to 
verify  totals. 

Such  a  card  system  could  easily  be  adapted  for  those  hospitals  which  utilize 
medical  records  personnel  for  completing  data  other  than  PSRO  review  infor- 
mation.   The  review  coordinator  could  note  the  date  the  abstract  (or 
worksheet)  was  sent  to  medical  records  for  completion  of  coding  and  then 
place    a  card  in  a  section  for  medical  records. 
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EXHIBIT  1 1- 5 
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EXHIBIT  II-6 


REVIEW  WORK  CONTROL  CARD  FILE 
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III.     AUDIT  PROCEDURES 


1.  PURPOSE 

Audit  procedures  are  periodic  activities  conducted  only  by  PSRO 
personnel,  or  by  experts  it  retains,  to  assess  the  effectiveness  of 
ongoing  data  quality  control  procedures,  to  determine  the  quality  of 
data  collected  by  individual  hospitals,  to  assess  personnel  staffing 
capabilities  and  needs,  and  to  aid  in  the  overall  management  of  the 
PSRO  program.     It  is  recommended  that  the  results  of  these  activities 
be  communicated  to  individual  hospitals  and  used  as  criteria  for 
continued  delegation  of  review  activities. 

Audit  activities  are  conducted  by  the  PSRO  to  fulfill  its  responsibility 
to  assess  and  monitor  hospital  performance  and  compliance  with  the 
Memorandum  of  Understanding.    Data  accuracy,  reliability  and  timeliness 
are  major  requirements  for  compliance  with  PSRO  standards.  Combining 
these  audit  procedures  for  data  quality  with  other  monitoring  activities 
may  be  desirable  in  many  PSROs. 

Audit  procedures  can  be  both  detailed  and  general,  depending  upon  the 
functional  area  under  audit.    For  instance,  in  the  functional  area  of 
data  abstracting  and  recording,  which  was  subdivided  into  five  components 
in  the  previous  chapter,  there  is  one  audit  procedure  covering  all  five 
components.     The  same  is  generally  the  case  for  the  other  functional 
areas;  that  is  periodic  audits  for  large  groupings  of  data  will  be 
conducted. 

Some  audit  procedures  will  be  used  more  frequently  than  others,  because 
they  can  be  done  quickly  and  easily.     Other  procedures  requiring  more 
time  (such  as  complete  re-abstracting)  may  be  conducted  at  each  hospital 
as  infrequently  as  every  six  months  to  a  year. 

As  in  the  previous  chapter,  audit  procedures  will  be  presented  by  each 
functional  area  with  a  background  section  describing  the  quality  require- 
ments, and  questions  which  should  be  answered  prior  to  conducting  the 
audit  procedure. 

2.  DATA  ABSTRACTING  AND  RECORDING 
2.1  Background 

Ongoing  operational  controls  described  in  the  previous  chapter 
included: 

-    Patient  Identification  Review  in  which  a  daily  verification 
utilizing  an  independent  data  source  takes  place 
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-  Pre-Screening  of  all  abstracts  for  completeness,  legibility, 
and  logic  prior  to  data  entry 

-  Focused  Pre-Screening  on  all  abstracts  from  a  selected 
hospital  for  selected  errors  (as  determined  by  PSRO  analysis) 

-  Determination  and  Elimination  of  Ordering  Discrepancies,  and 

-  Verification  of  Coding  Accuracy,  both  of  which  requires  re- 
abstracting  five  to  seven  records  for  diagnoses  and  procedures 
only.    Prior  analysis  of  diagnostic  and  procedural  errors 
determines  selection  of  abstracts  to  be  verified  and  medical 
records  to  be  pulled.    These  two  procedures  would  be  per- 
formed regularly  but  not  as  frequently  as  the  above  procedures. 

-  Bill  Certification  Review  in  which  five  to  seven  bills  are 
pulled  by  Billing  Department  and  compared  with  abstracts 
randomly  selected. 

In  addition  to  determining  the  effectiveness  of  the  above 
operational  controls,  audit  procedures  for  data  abstracting 
and  recording  should  also  assess  the  quality  of  review  documen- 
tation and  timeliness  of  abstract  completion  and  submittal. 
In  addition  to  determining  accuracy  of  individual  data  elements, 
such  as  diagnoses  and  procedures  and  those  errors  found 
through  analysis  of  reports,  audit  procedures  should  result  in 
an  accuracy  rate  determination  for  individual  employees  and 
hospitals. 

The  following  audit  procedure  covers  all  aspects  of  data 
abstracting  and  recording  with  the  exception  of  patient 
identification  review. 

2 . 2    Audit  Procedure:  Re-Abstracting 

Re-abstracting  is  the  process  of  completing  a  new  abstract 
form  (data  input  form)  utilizing  the  completed  medical  record. 
Re-abstracting  is  done  by  a  PSRO  employee  other  than  the  person 
who  completed  the  original  abstract.     The  new  abstract  is  then 
compared  to  the  original  one  (or  a  computer  printout  of 
the  abstract)  and  discrepancies  noted.     For  some  discrepancies 
it  must  further  be  determined  which  abstract  is  correct.  For 
instance,  on  the  Diagnoses  and  Procedures  Verification  Form 
(Exhibit  II-4) ,  a  column  has  been  provided  for  checking  which 
are  correct  codes,  incorrect  codes  and  indeterminate  codes. 
Quite  often,  correct  codes  cannot  be  determined  due  to  incomplete 
medical  records  and  medical  terminology  which  does  not  exactly 
match  terminology  in  code  books.     In  determining  accuracy  rates 
for  each  data  field,  this  factor  would  have  to  be  taken  into 
consideration. 
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The  amount  of  time  required  to  re-abstract  a  single  medical 
record  will  vary  according  to  the  number  and  type  of  PSRO  data 
elements,  the  organization  and  quality  of  the  medical  record, 
the  medical  complexity  of  the  case  being  investigated,  the 
familiarization  of  the  abstractor  with  hospital  and  medical 
records  department  procedures,  and  with  the  training  and 
background  of  the  abstractor. 

Once  the  PSRO  determines  how  many  and  which  abstracts  are  to 
be  re-abstracted,  it  must  notify  the  hospital,  giving  it  enough 
time  to  pull  the  required  records.     To  the  extent  possible, 
re-abstracting  should  be  arranged  so  as  not  to  cause  a  major 
disruption  in  the  activities  of  the  medical  records  department. 

Sample  selection  procedures  should  be  developed  to  identify 
abstracts  which  are  most  representative  of  possible  problem 
areas  yet  provide  a  good  cross-section  of  abstracts  completed 
by  various  individuals.     The  size  of  the  sample  will  depend 
upon  the  hospital's  volume  of  Federal  beneficiaries  discharged. 
The  size  of  a  sample  may  also  vary  as  a  function  of  previous 
findings  and  audit  experience. 

The  New  York  County  Health  Services  Review  Organization  (NYCHSRO) 
has  developed  a  sophisticated  sampling  technique  for  selection  of 
sample  sizes,  which  is  described  briefly  in  Appendix  C.    A  very 
intricate  statistical  sampling  method  involving  proportions  of 
items  in  error  and  confidence  limits  for  data  was  used  to  arrive 
at  the  sample  sizes.     NYCHSRO  believed  such  a  method  was  necessary 
because  of  its  tremendous  discharge  volumes.     Statistical  sampling 
methods,  however,  are  probably  not  necessary  for  the  majority  of 
PSROs.    Whatever  sample  is  selected,  it  should  contain  not  only  data 
expected  to  be  defective,  but  also  a  moderate  amount  of  abstracts 
which  have  passed  all  previous  screens  (including  computer  edits) 
and  are  believed  to  be  acceptable.     Generally,  the  sample  should 
neither  be  so  large  that  it  would  require  excessive  employee  time, 
nor  so  small  that  it  would  give  a  misleading  picture  of  the  level 
of  quality. 

Once  a  sample  has  been  selected,  and  abstracts  retrieved  from 
PSRO  files,  the  PSRO  will  notify  the  hospital  in  writing, 
listing  the  identifying  information  necessary  for  the  hospital 
to  pull  the  medical  records.  1/ 


1/  PSROs  with  hospitals  using  an  abstract  service  where  there  are  no  hard 
copy  abstracts  available  should  request  individual  patient  printouts 
from  the  hospital  or  the  data  collector  and/or  processor.     If,  for  any 
reason,  this  cannot  be  accomplished,  the  review  coordinator's  supporting 
documentation  (i.e.,  worksheet)  would  have  to  be  used.     If  there  is  no 
supporting  documentation  and  computer  listings  are  not  available,  the 
PSRO  will  not  be  able  to  audit  the  review  process  adequately. 
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Comparison  of  samples  could  be  taken  one  step  further  in  those 
PSROs  with  computer  capability  to  produce  printouts  of 
selected  abstracts.     The  printout,  hard  copy  abstracts,  re- 
abstracts  and  source  documents  (medical  record  and  review 
worksheet)  could  also  be  compared  for  discrepancies. 

After  comparing  abstracts,  the  PSRO  should  develop  accuracy 
rates  for  each  data  element.     These  accuracy  rates  will  help 
determine  credibility  of  reports,  profiles,  and  norms  developed 
by  the  PSRO.     The  use  of  the  data  will  determine  the  significance 
of  each  element.    Among  PHDDS  elements,  the  impact  of  errors 
in  some  elements  may  be  much  greater  than  in  other;  i.e.,  diagnoses 
in  the  development  of  length  of  stay  norms  versus  race.  Similar 
analysis  should  be  conducted  for  each  hospital. 

Types  of  errors  should  be  classified:     such  information  is 
useful  in  determining  training  and  retraining  needs.     It  may 
also  indicate  a  need  to  change  procedures,  data  forms  or 
quality  control  procedures. 

Coding  Errors  Which  May  Be  Encountered 

1.  Ordering  Procedures-Abstract  Service:     difference  between 
definitions  of  the  PSRO  and  abstract  service 

2.  Ordering  Procedures-Hospital:     difference  between  hospital 
definitions  and  PSRO  definitions 

3.  Ordering  Judgment:     differences  in  opinion  in  inter- 
preting medical  record 

4.  Coding  Procedure:     discrepancy  caused  by  systematic  misuse 
or  misunderstanding  of  coding  system  (i.e.,  use  of 
alphabetical  index  alone) 

5.  Coding  Judgment:     difference  of  opinion  in  classifying 
a  diagnosis  not  specifically  listed  in  coding  books 

6.  Dependent:     selection  of  principal  procedure  dependent  on 
principal  diagnosis  selection 

Abstracting  and  Recording  Errors 

1.     Omission  -  data  element  left  blank 
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2.  Transcriptional  -  information  incorrectly  copied  from 
source  document 

3.  Procedural  -  data  coded  incorrectly;  i.e.,  alpha  and 
numeric  codes  used  incorrectly,  fields  not  properly  right 
or  left  justified,  review  procedures  not  properly  followed 
causing  abstract  error. 

In  those  PSROs  with  data  quality  control  monitoring  separate 
from  the  PSRO  hospital  assessment  and  monitoring  of  review 
activities,  results  of  re-abstracting  should  be  shared. 
Errors  made  in  the  review  process  causing  data  element  errors 
should  be  brought  to  the  attention  of  the  PSRO  Review 
Coordinator  Supervisor  or  other  person  responsible  for 
concurrent  review  monitoring. 

During  the  re-abstracting  process  copies  of  the  hospital  bills 
sent  to  the  fiscal  intermediary  should  be  reviewed  to  verify 
that  all  bills  were  certified  and  that  the  correct  number  of 
days  were  certified.     The  PSRO  logs  containing  that  number  of  bills 
certified  for  the  last  quarter  should  be  compared  to  the  fiscal 
intermediary's  listings  of  Federal  patients  if  these  were  made 
available  to  PSROs. 

3.     FORMS  FLOW  CONTROL  AND  DATA  TRANSFER 

As  part  of  the  periodic  data  quality  audit,  the  PSRO  needs  to  examine 
the  integrity  of  data  transfer  procedures.  There  are  essentially  two 
elements  in  the  accomplishment  of  this  objective: 

The  PSRO  should  verify  the  accuracy  of  statistics  maintained  by 
hospitals,  intermediaries,  data  collectors  and/or  processors,  and 
the  PSRO  itself  concerning  the  numbers  of  admissions  and  discharges. 
The  particular  manner  in  which  this  can  be  accomplished  is  highly 
dependent  upon  the  PSRO's  individual  system  configuration.  For 
example,  the  PSRO  may  suggest  or  require  that  review  coordinators 
maintain  a  notebook  of  the  hospital's  daily  discharge  listings. 
The  PSRO  audit  team  could  then  compare  cumulative  listings  from 
the  notebook  with  statistics  derived  from  the  PSRO  data  receipt  log, 
processor  reports  and,  possibly,  statistics  obtained  from  fiscal 
intermediaries.     In  this  way,  the  PSRO  could  identify  potential 
trouble  spots  in  the  system. 

The  effectiveness  of  any  quality  control  programs  depends  in  large 
part  on  the  extent  to  which  personnel  adhere  to  procedures. 
Uniformity  of  activity  requires  that  procedures  be  documented  in  a 
useful  form.     The  PSRO  audit  team  can,  therefore,  use  such  procedure 
manuals  in  conjunction  with  team  activities  to  determine  the  extent 
to  which  program  personnel  adhere  to  procedures.    Deviations  should 
be  documented  and  appropriate  corrective  action  taken. 
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4.     DATA  ENTRY  AND  EDIT 


Data  entry  and  edit  auditing  is  a  key  part  of  maintaining  control  over 
the  quality  of  systems  operations.    Activities  that  address  this  area 
include  the  following: 

The  audit  team  should  scrutinize  monthly  and  quarterly  error  reports 
against  source  documents  to  determine  key  entry  and  programming 
errors . 

Procedure  manuals  should  be  reviewed  for  completeness,  and  the 

audit  team  should  attempt  to  examine  the  extent  to  which  such  manuals 

are  followed. 

In  the  case  of  an  on-site/on-line  CRT  system,  the  team  should  examine 
entry  logs  and  system  performance  logs  to  determine  any  patterns  or 
trends  that  require  attention,  such  as  increasing  key  entry  times 
or  significant  system  down  time. 

5.     DATA  PROCESSING  AND  OUTPUT  ANALYSIS 

A  final  step  in  the  control  of  the  data  system  entails  checks  on  the 
system  edit  package  capabilities  and  the  output  reports  themselves. 

Prior  to  system  implementation,  the  PSRO  should  subject  the 
processor's  edits  to  a  test  package  of  dummy  abstracts.  These 
abstracts  contain  coding  and  logic  errors  known  to  the  PSRO.  The 
error  reports  generated  by  the  processor  gives  the  PSRO  a  good  idea 
of  the  nature  and  adequacy  of  the  processor's  edit.     HSQB  has 
available  such  a  package  designed  to  test  PHDDS  edits. 

Periodically,  the  PSRO  should  submit  test  packages  to  check  the 
ongoing  quality  of  the  processor's  system.    Unless  problems  are 
detected,  such  tests  do  not  need  to  be  as  exhaustive  as  the  initial 
system  test.    Periodic  tests  provide  the  PSRO  with  the  opportunity 
for  examining  the  adequacy  with  which  the  processor  has  implemented 
changes  in  the  system  that  may  have  been  requested  by  the  PSRO. 

In  addition  to  the  test  package,  the  audit  team  should  examine 
output  reports  for  logic.     The  particular  logic  check  or  manual 
relationship  edits  to  be  performed  should  be  documented  in  the  audit 
team  procedures  manual. 
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APPENDIX  A 


CRITERIA  DEVELOPMENT 
FOR  DATA  QUALITY  MONITORING 


Introduction 


In  many  instances  it  is  possible  to  examine  the  quality  of  PSRO  data  from  analysis 
of  the  routine  reports  produced  by  PSRO  data  processors.    This  is  particularly 
true  with  respect  to  areas  of  coding  difficulty,  which  have  historically  been  both 
tedious  and  difficult  to  identify  and  control. 

The  following  materials  produced  by  the  Commission  on  Professional  and  Hospital 
Activities  (CPHA)  are  strongly  oriented  in  this  direction  and  provide  the 
groundwork  for  data  quality  control  activities  which  may  be  performed  to  a  con- 
siderable extent  within  the  PSRO  offices.    PSROs  are  encouraged  to  pursue  such 
quality  control  procedures,  which  may  prove  both  efficient  and  effective  in 
accomplishing  the  objectives  of  data  quality  control  activities.    It  must  be 
understood,  however,  that  the  approach  is  still  being  developed  and  refined  by 
CPHA.     Criteria  to  be  used  for  examining  routine  reports  and  guidelines  for  use 
of  these  criteria  in  an  ongoing  quality  control  system  are  being  developed  by 
CPHA  and  will  be  published  and  available  to  PSROs. 
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Criteria  Development  for  Data  Quality  Monitoring* 
Any  quality  control  system  has  five  essential  components: 

1.  Establish  criteria. 

2.  Perform  surveillance. 

3.  Analyze  deviations  of  performances  from  criteria. 

4.  Take  action. 

5.  Follow  up 

Monitoring  is  one  method  of  surveillance  which,  when  used  with  specified 
criteria,  is  useful  for  identifying  chronic  or  systematic  errors 
occurring  in  a  process.     A  criterion  is  a  parameter  and  a  standard  taken 
together — a  parameter  is  something  which  can  be  measured  and  a  standard 
is  the  expected  level  of  performance  for  that  parameter.     For  example, 
in  monitoring  assignment  of  principal  diagnosis  a  parameter  might  be 
"percent  of  patients  for  which  H-ICDA**  code  412.9  (Other  and  unspecified 
chronic  ischemic  heart  disease)  is  recorded  as  principal  diagnosis"  and 
the  standard  would  be  "07,."     The  criterion  then  for  H-ICDA  code  412.9 
appearing  as  principal  diagnosis  is  07,  of  all  patients. 


*This  concept  is  excerpted  from  a  forthcoming  publication  Quality  Control  of 
Diagnosis  and  Procedure  Coding  to  be  published  by  the  Commission  on 
Professional  and  Hospital  Activities,  Ann  Arbor,  Michigan. 

**Hospital  Adaptation  of  ICDA  (H-ICDA),  Second  Edition,   Commission  on 
Professional  and  Hospital  Activities,  Ann  Arbor,  Michigan,  1973. 


cm 


Commission  on  Professional  and  Hospital  Activities     196S  Green  Road  Ann  Arbor,  Michigan  48105 

in  Coilei 

A-2 


Monprofit  organization  sponsored  by  the  American  College  of  Physicians,  American  College  of  Surgeons,  American  Hospital  Association,  Southwestern  Michigan  Hospital  Counci\ 
CPHA  86-9-73 


One  Method  of  data  accuracy  monitoring  is  comparing  the  data  in  routine 
statistical  reports  and  patient  indexes  with  pre-established  criteria  to 
check  for  the  accuracy  of  coding  diagnoses  or  procedures,  of  sequencing 
of  diagnoses  or  procedures,  or  of  abstracting.    Deviations  of  performance 
from  criteria,  as  shown  by  the  monitor,  indicate  the  need  for  further 
investigation  of  the  cause  of  the  errors. 

Data  quality  monitoring  finds  conditions  which  could  appear  logical  on  a 
single  abstract  but,  when  found  among  several  abstracts,  indicate  chronic 
problems;  for  example,  problems  in  the  training  of  coders  or  misunder- 
standing of  the  procedures  used  to  code. 

Establishing  criteria  for  monitoring  is  a  two-step  procedure: 

1.  Perform  an  analysis  to  determine  the  potential  sources  of 
systematic  error  which  can  occur. 

2.  Establish  criteria  which  effectively  detect  these  systematic 
errors . 

Attachments  1  and  2  illustrate  this  two-step  procedure  for  establishing 
criteria  for  accuracy  of  diagnosis  coding  and  for  assignment  of  principal 
diagnosis.    Examples  of  specific  criteria  are  given  for  each  source  of 
error.    Many  other  criteria  can  be  developed.    Attachment  3  discusses  the 
sources  of  systematic  errors  for  diagnosis  coding  and  for  assignment  of 
principal  diagnosis. 
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It  should  be  noted  that  monitoring  can  be  done  using  only  a  small  number  of 
criteria.    Because  the  procedures  a  coder  uses  are  probably  the  same  for 
all  codes ,  the  types  of  errors  uncovered  by  using  a  few  criteria  pervade 
the  coding  of  all  diagnoses. 

Additional  criteria  for  diagnosis  coding  and  sequencing  and  criteria  for 
operation  coding s  operation  sequencings  and  abstracting  of  other  items  in 
the  PHDDS  9  are  being  developed  and  will  be  included  in  the  CPHA  publication. 

Attachments ;    1.    Assigning  a  Diagnosis  Code 

2o    Assignment  of  Principal  Diagnosis 
3.     Sources  of  Systematic  Errors 
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Examples  of  Criteria  for  Error  Type  T: 
Coder  Chooses  Incorrect  Terminology* 


Criteria  1: 

Parameter:    %    of  patients  with  code  410.9  (Acute  myocardial  infarction, 
site  not  specified)  who  stayed  more  than  one  day. 

Standard:  0% 

Criteria  2: 

Parameter:    Number  of  patients  with  code  583  (Glomerulonephritis, 
unqualified) . 

Standard:  0 

Criteria  3: 

Parameter:    Number  of  patients  with  code  59Q.2  (Pyelonephritis, 
unspecified) . 

Standard:  0 

Criteria  4: 

Parameter:  Number  of  patients  with  code  490  (Bronchitis,  unqualified). 

Standard :  0 
Criteria  5: 

Parameter:  Number  of  patients  with  code  725.3  (Derangement  and 

displacement  of  intervertebral  disc,  site  unspecified). 

Standard:  0 


*Criteria  are  based  on  classification  structure  of  the  Hospital  Adaptation 
of  ICDA  (H-ICDA),  Second  Edition. 
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Examples  of  Criteria  for  Error  Type  C: 
Coder  Chooses  Incorrect  Code  When  Terminology  Is  In  Index* 


> 


Criteria  1: 


Parameter: 


Parameter: 

Standard: 
Criteria  3: 

Parameter: 


%  of  patients  with  code  412.0  (Chronic  ischemic  heart 
disease  with  healed  myocardial  infarction)  who  also  have 
code  412.9. 


Standard:  0% 
Criteria  2: 


%  of  patients  with  any  code  in  the  range  43Q.Q-436.1  or 

code  438.0  (Cerebrovascular  disease)  with  a  code  in  the 
range  of  344.0-344.9. 

0% 


%  of  patients  with  code  650  (Delivery  without  mention 
of  complication)  who  also  have  a  code  in  the  range  of 
651.0-664.9  (Delivery  with  complication). 


Standard:  0% 


*Criteria  are  based  upon  the  coding  rules  of  the  Hospital  Adaptation  of  ICDA 
(H-ICDA),  Second  Edition. 
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Examples  of  Criteria  for  Error  Type  A: 
Additional  Code  Required  But  Not  Present* 


Criteria  1: 


Parameter:     %  of  patients  having  code  250.2  (Adult  onset  diabetes 

me  1 lit us  with  other  complications)  with  additional  codes 
indicating  the  complications,  e.g.,  codes 


357.9 
367.0 
377.4 
445.9 


707.0 
707.1 
707.9 


Standard : 


100% 


Criteria  2; 


Parameter:     %  of  patients  having  code  305.3  (Chronic  nonpsychotic 

organic  brain  syndrome  due  to  circulatory  disturbances) 
with  additional  codes  indicating  the  cause  (codes  in 
the  range  of  393-458.9). 


Standard: 
Criteria  3: 

Parameter: 


100% 


%  of  patients  with  code  403.2  (Hypertension  secondary  to 
other  renal  disease)     with  additional  codes  indicating 
the  cause,  e.g.,  codes 


250.6 
580-593.5 
753.0-753.5, 
753.9 


Standard: 


100% 


*Criteria  are  based  upon  the  coding  rules  of  the  Hospital  Adaptation  of  ICDA 
(H-ICDA),  Second  Edition. 
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Examples  of  Criteria  for  Error  Type  P: 
Error  Made  in  Assignment  of  Principal  Diagnosis* 


Criteria  1; 

Parameter:    %  of  patients  with  principal  diagnosis  of  412.9  (Other 
and  unspecified  chronic  ischemic  heart  disease). 

Standard:  0% 

Criteria  2: 

Parameter:    %  of  patients  with  principal  diagnosis  of  437 

(Generalized  ischemic  cerebrovascular  disease) . 

Standard:  0% 

Criteria  3: 

Parameter:    %  of  patients  with  principal  diagnosis  of  412.0 

(Chronic  ischemic  heart  disease  with  healed  myocardial 
infarction) . 

Standard:  0% 


♦Criteria  are  based  upon  the  UHJ3DS  definition  of  principal  diagnosis  and 
the  Hospital  Adaptation  of  ICDA  (H-ICDA) ,  Second  Edition. 


2-2 


CM 

Commission  on  Professional  and  Hospital  Activities     1968  Green  Road  Am  Arbor,  Michigan  48105 

Nonprofit  organization  sponsored  by  the  American  College  of  Physicians,  American  College  of  Surgeons,  American  Hospital  Association,  Southwestern  Michigan  Hospital  Council 


Attachment  3 


Sources  of  Systematic  Errors 


The  processes  of  coding  diagnoses  and  procedures  and  of  assigning 
principal  diagnosis  and  principal  procedure  are  complex  and  errors  can 
occur  at  any  one  of  a  number  of  points. 

The  sources  of  error  displayed  in  Attachments  1  and  2  describe  the  major 
categories  of  systematic  error  which  occur  in  diagnosis  coding  and  in  the 
assignment  of  principal  diagnosis. 

Each  of  these  major  categories  of  error  may  have  more  than  one  underlying 
cause.    When  this  occurs  it  is  not  possible  to  conclude,  by  use  of  the 
criteria  alone,  just  what  the  underlying  cause  might  be.  Accurate 
conclusion  here  is  dependent  upon  observation  of  the  actual  procedures  used 
by  the  coder. 

The  following  is  a  description  of  the  four  sources  of  systematic  error  in 
diagnosis  coding  and  in  assignment  of  principal  diagnosis  which  can  be 
detected  by  means  of  a  data  monitor.    When  a  category  of  error  has  more 
than  one  underlying  cause,  this  is  also  discussed. 

1.  (T)     Coder  chooses  incorrect  terminology. 

The  first  step  in  coding  is  determination  of  the  terms  to  be  coded. 
Errors  occur  when  the  coder  selects  the  incorrect  or  incomplete 
terminology  so  that  the  code  assigned  to  those  terms  Cwhich  may  be 
the  correct  code  for  those  terms)  does  not  describe  the  patient's 
actual  diagnosis  or  condition. 

This  type  of  error  is  usually  caused  by  the  coder  using  only  the 
face  sheet,  which  itself  may  contain  incomplete  terminology;  to 
determine  the  terminology  to  be  coded.    Accurate  terminology  determination 
requires  the  entire  record  be  reviewed. 

Coding  by  use  of  the  face  sheet  only  can  result  in  errors  in  both 
individual  cades  as  well  as  errors  in  the  totality  of  codes. 

2.  (C)    Coder  chooses  incorrect  code. 

Many  categories  in  the  disease  classification  (both  H-ICDA-2  and 
ICDA-8*)  contain  instructions  telling  the  user  that  certain  codes  may  not 
be  used  in  combination  with  certain  other  codes. 


*Eighth  Revision  International  Classification  of  Diseases,  Adapted  for 
Use  in  the  United  States  (ICDA-8). 
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Attachment  3,  Sources  of  Systematic  Errors  (continued) 


Errors  occur  when  the  coder  does  not  follow  the  instructions  of  the 
classification  and  uses  incorrect  code  combinations,  either  because  of 
coding  directly  from  the  index  (which  does  not  contain  the  instructions) 
or  because  even  though  the  Tabular  List  is  referred  to  the  instructions 
are  not  read  (or  understood)  completely. 

3c     (A)     Additional  code  required  but  not  present. 

Many  categories  in  the  disease  classification  (both  H-ICDA-2  and  I CPA- 8) 
contain  instructions  telling  the  user  that  certain  conditions  require 
more  than  one  code  in  order  to  describe  the  condition  completely. 

Errors  occur  when  the  coder  does  not  follow  the  instructions  of  the 
classification  and  omits  the  additional  code.    This  occurs  when  the 
coder  either  codes  directly  from  the  index  (which  does  not  contain  the 
instructions)  or  refers  to  the  Tabular  List  but  does  not  read  (or  under- 
stand) the  instructions  completely. 

4.     Error  in  assignment  of  principal  diagnosis. 

Error  in  assignment  of  principal  diagnosis  has  four  causes: 

a.  The  abstractor  has  a  misunderstanding  of  the  definition  of 
principal  diagnosis. 

b.  The  abstractor  makes  an  attempt  to  determine  which  of  many 
diagnoses  is  principal,  and  though  a  decision  is  based  on 
information  gleaned  from  the  entire  medical  record,  the 
wrong  decision  is  made. 

c.  The  abstractor  makes  an  attempt  to  determine  which  of  many 
diagnoses  is  principal  and  bases  a  decision  on  the  infor- 
mation contained  on  the  face  sheet  of  the  medical  record. 

d.  The  abstractor  accepts  without  question  the  sequence  of 
diagnoses  as  written  by  the  physician  on  the  face  sheet  of 
the  medical  record  and  the  physician  has  not  sequenced  the 
diagnoses  according  to  the  requirements  of  the  UHDDS. 

The  monitor  criteria  detect  errors  in  the  selection  of  principal 
diagnosis  but  do  not  detect  the  underlying  cause.    This  can  be 
done  only  by  observing  the  actual  procedure  used  by  the  abstractor. 
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APPENDIX  B 


CASE  STUDY 
CENTRAL  MARYLAND 
PROFESSIONAL  STANDARDS  REVIEW  ORGANIZATION 


Preface 


The  following  case  study  describes  the  data  quality  control  procedures 
developed  by  Central  Maryland  Professional  Standards  Review  Organization 
(CMPSRO)  to  control  the  quality  of  data  collected  through  existing  dis- 
charge abstract  organizations. 

We  are  grateful  to  CMPSRO  and  Barbara  Reinthala,  Health  Data  Analyst,  for 
providing  us  with  the  procedures  and  forms  which  CMPSRO  is  using  and  in 
the  process  of  further  refining  to  control  the  quality  of  the  data. 
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CENTRAL  MARYLAND  PSRO  PROCEDURES 


Information  collected  on  Federal  beneficiaries  via  the  HDS  and  PAS 
abstracting  systems  are  routinely  monitored  for  accuracy  on  a  monthly 
basis.     The  number  of  records  monitored  is  based  on  the  number  of  work- 
sheets received  at  the  CM-PSRO  offices  for  each  hospital  during  a 
designated  month.     Usually,  ten  percent  of  a  month's  discharges  will  be 
monitored.     This  may  vary  according  to  the  size  of  the  hospital. 

SITE   (MONITORING)  PREPARATION 

Hospitals  to  be  monitored  are  notified  at  least  one  week  prior  to  a 
visit.     If  a  visit  is  cancelled  or  has  to  be  rescheduled,  hospitals  must 
be  notified  of  this  as  soon  as  possible. 

A.  Determine  which  month's  abstracts  are  to  be  monitored. 

B.  Determine  the  number  of  abstracts  received  from  the  hospital  during 
that  month  and  percentage  to  be  checked. 

C.  Randomly  select  and  copy  (from  batches  on  file)  worksheets  which 
are  to  be  monitored,  if  necessary. 

D.  Contact  the  designated  hospital  and  provide  medical  records  personnel 
with  the  names,  medical  record  numbers,  and  discharge  dates  of 
patients  whose  records  are  to  be  reviewed. 

ITEMS  TO  BE  MONITORED 

HDS/PAS  abstracts  are  to  be  checked  for: 

A.  Accuracy  of  information  recorded  on  the  abstracts:  U"  elements 
(items  1  through  14) .  Abstracts  should  be  verified  by  comparing 
information  in  the  medical  record  with  that  on  the  abstract. 

B.  Accuracy  of  coding  -  Discharge  diagnoses  and  procedures  should  be 
verified  by  checking  the  medical  record  and  by  consulting  H-ICDA-2 
or  ICDA-8  codebooks. 

C.  Accuracy  of  concurrent  review:      P"  elements  (items  15-31).  The 
concurrent  review  section  of  the  corresponding  worksheet  should  be 
compared  with  the  concurrent  review  portion  of  the  HDS/PAS  abstracts 
to  assure  that  information  has  been  correctly  transferred  from  the 
worksheet  to  the  abstract. 

RECORD/ABSTRACT  REVIEW 

Data  monitoring  worksheets  are  used  when  reviewing  HDS/PAS  abstracts. 
Listed  on  the  worksheets  are  the  31  data  elements  that  are  currently 
being  collected  by  CM-PSRO.     Under  the  columns  opposite  the  data  elements, 
record  for  each  record/abstract  reviewed:     (see  Exhibit  B-l) 
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Date 


Exhibit  B-l 


DATA  ELEMENT 


RECORD// 

RECORD// 

RECORD// 

RECORD// 

RECORD// 

RECORD// 

RECORD  //I 

RECORD// 

ERROR 

ERROR 

ERROR 

ERROR 

ERROR 

ERROR 

ERROR 

ERROR 





 — 

1.  Person  ID 

2.  DOB 

3.  Sex 

4.  Race 

5.  Residence 

6.  Hospital  ID 

7.  Admit  Date/Hour 

8.  Disch.  Date 

9.  Attending  MD 

10.  Operating  MD 

11.  DX/Coding 

12.  Procedures 

13.  Pt.  Disposition 

14.  Exp.  SRC  Pay  M. 

15.  ICD 

16.  ACP 

17.  BICD 

18.  ACLR 

19.  TDC 

20.  TR-PA 

21.  Total  CSRs 

22.  Ext.  Denial 

23.  Days  1st  CSR 

24.  Total  LOS 

25.  Acute  Cert.  Days 

26.  Admin.  Cert.  Days 

27.  Grace  Days 

28.  Other  Days 

29.  Admin.  Reasons 

30.  Other  Reasons 

31.  Date  Cert. /Rev. 

Total  Errors 


A.      The  medical  record  number. 


B.      The  correct  information  when  an  error  is  found. 

If  there  are  no  errors  for  a  particular  element,  leave  the  space  blank. 
One  column  should  be  used  for  each  record/abstract  reviewed.     The  total 
number  of  errors  found  for  each  record  should  be  noted  under  the  "Record 
Error"  columns.     At  the  end  of  each  visit,  a  copy  of  the  data  monitoring 
worksheet  should  be  made  and  attached  to  abstracts  for  which  errors  have 
been  detected.     These  should  be  given  to  the  appropriate  person  in  the 
Medical  Records  Department  for  abstract  correction. 

MONTHLY  DATA  MONITORING  SITE  VISIT  REPORT 

At  the  end  of  each  site  visit,  the  short  form  data  monitoring  report 
should  be  completed.     These  reports  should  be  completed  as  shown  below: 
(see  Exhibit  B-2) 


Note:     The  calculations  for  the  error  rate  is  based  on  number  of  elements 
(31)  times  the  number  of  records  reviewed.     One  Data  Monitoring  Site 
Visit  Report  sheet  may  be  used  for  three  visits;  however,  to  avoid 
possible  errors,  the  same  sheet  should  be  used  for  one  hospital.  Data 
Monitoring  Worksheets  should  be  attached  to  the  report  sheet  and  used 
as  a  reference  for  preparing  the  quarterly  report  to  the  hospitals. 

QUARTERLY  SUMMARY 

At  the  end  of  each  quarter  (or  three-month  period) ,  a  summary  of  the 
number  and  types  of  errors  found  will  be  presented  to  administrative 
liaisons  and  to  the  Medical  Records  Department  heads.  A  copy  of  the 
summary  will  be  kept  on  file  at  the  CM-PSRO  offices. 


Hospital: 

Number  of  Discharges: 
Number  of  Records  Reviewed: 


800 
300 
30 


Date  Reviewed:  10/28/76 
Discharge  for  Month  of:  Sept. 
Reviewed  by :     S .  Smith 


Number  of  errors  found: 
Error  Rate: 

Person  notified  of  errors: 


36 
11% 

Mrs.  J.  Jones 
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DATA  MONITORING  SITE  VISIT  REPORT 


Exhibit  B-2 


Hospital: 

Number  of  Discharges: 
Number  of  Records  Reviewed: 


Date  Reviewed: 
Discharges  for  month  of: 
Reviewed  by: 


Number  of  Errors  found; 
Error  rate: 

Person  notified  of  errors: 


COMMENTS 


Hospital: 

Number  of  Discharges: 
Number  of  Records  Reviewed: 


Date  Reviewed: 
Discharges  for  month  of: 
Reviewed  by: 


Number  of  Errors  found: 
Error  rate: 

Person  notified  of  errors; 


COMMENTS 


Hosp ital: 

Number  of  Discharges: 
Number  of  Records  Reviewed: 


Date  Reviewed; 
Discharges  for  month  of: 
Reviewed  by: 


Number  of  Errors  found: 
Error  rate: 

Person  notified  of  errors: 


COMMENTS 


EXHIBIT  B-3 


Total 
Def. 

Date 

Cert. 

« 

V 

Tot. 
Rev. 

td 

01 

CSR 

D 

BICD 

TDC 

Den. 
Used 
Days 

Grace 

Adm. 
Day  8 

A-C 
Days 

• 

4J  to 
o  o 

H  J 

0S 

CO  < 

CSR 

OS 
H 

PA 

tj  tc 
cd  tn 

A.CLR 

ICP 

ACP 

Dls. 
Date 

Adm. 
Date 

SSI 

HA 

MDC 

Count 

BATCH 
DATE 

- 

Totals 

01 
Ai 
C3 

Worksheet  Deficiency  Monitoring 


PURPOSE:     To  determine  whether  PHDDS  elements  are  recorded  consistently 
and  accurately 

Ten  percent  of  the  worksheets  contained  in  batches  received  from  CM-PSRO 
hospitals  will  be  reviewed  for  deficiencies  (errors) .     One  CM-PSRO 
Worksheet  Deficiency  Form  per  hospital  per  month  will  be  maintained  for 
recording  errors  found  during  review.    All  batches  received  from  a  given 
hospital  within  a  one-month  period  will  be  recorded  and  reviewed. 

If  a  hospital  is  found  to  be  making  errors  frequently,  a  100  percent 
review  of  documents  received  from  that  hospital  will  be  performed  for  a 
specified  period  of  time  (usually  one  month) .     The  Patient  Review 
Coordinator  office  will  then  be  contacted  to  be  informed  of  the  results 
of  the  review  and  to  discuss  the  cause (s)  of  the  errors.     If  this  does 
not  alleviate  the  problem,  the  PRC(s)  and  clerk-technician,  if  any,  will 
be  requested  to  attend  a  short  refresher  training  session  on  data 
collection  and  recording.     The  training  session  will  be  conducted  by 
the  Systems  Manager  and/or  the  Health  Data  Analyst.     The  CM-PSRO  Worksheet 
Deficiency  Form  should  be  completed  in  the  following  manner:     (Exhibit  B-3) 

Hospital  :        Enter  the  PSRO  assigned  number  of  the  hospital. 

Month  :        Enter  the  month  batches  were  received. 

Batch  Date  ;        Record  the  date  of  receipt  and  batch  letter  (if 

any)  noted  on  the  batch  envelope. 

Count  :        Record  the  total  number  of  documents  contained  in 

the  batch. 

MA/MDC  :        Determine  whether  the  Medicare  or  Medicaid  number 

has  been  recorded.     (note:     If  the  patient  is  a 
medical  assistance  applicant,  or  has  been  designated 
as  a  potential  MA/MDC  patient,  omission  of  this  number 
will  not  be  considered  as  an  error;  however,  letters 
indicating  the  potential  pay  source  should  be  circled) . 

SS#  :        Determine  whether  the  patient's  social  security  number 

has  been  recorded.     (note:     If  the  SS#  is  recorded  as 
unknown  or  the  patient  has  no  SS#,  omission  of  this 
item  will  not  be  considered  as  an  error) . 

Adm.  Date  :        Determine  whether  the  admission  date  has  been  recorded. 

Dis.  Date  :        Determine  whether  the  discharge  date  has  been  recorded. 
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Worksheet  Deficiency  Monitoring 


Determine  whether  the  admission  certification  process 
was  recorded  accurately,  i.e. ,  if  the  ACP  was 
recorded  as  code  2  (concurrent  admission  review- 
certification  granted) ,  the  date  of  initial  review 
for  certification  should  be  no  more  than  two  working 
days  after  the  patient's  admission  and  before 
discharge. 

Determine  whether  the  number  of  days  certified  at 
admission  was  recorded  accurately  by  comparing  the 
number  entered  for  this  item  with  the  first  number 
recorded  under  the  "Extended"  column  of  the  worksheet. 
(Note:     This  item  should  not  be  completed  for  ACP 
code  3  -  certification  denied,  or  code  4  -  concurrent 
admission  review  not  performed) . 

Determine  whether  the  admission  certification  level 
of  review  was  recorded  accurately. 

Determine  whether  days  recorded  as  being  assigned  at 
the  first  continued  stay  review  (if  any)  were  recorded 
accurately.     Comare  with  figure  given  in  "Extended" 
column  of  the  worksheet. 

Determine  whether  the  total  number  of  reviews  for 
admission  and  extension(s)  that  were  referred  to  the 
physician  advisor  were  recorded  accurately  (if 
applicable) . 

Determine  whether  the  total  number  of  approved 
continued  stay  reviews  performed  by  the  PRC  and  PA 
were  recorded  accurately. 

Determine  whether  the  number  of  extensions  (approved 
or  denied)  referred  to  the  PA  were  recorded  accurately. 

Determine  whether  the  total  length  of  stay,  computed 
from  date  of  admission  to  date  of  discharge,  was 
recorded  accurately.     Do  not  count  day  of  discharge 
when  figuring  this  item.     Total  LOS  =  Acute  Cert. 
Days  +  Grace  Days  +  Den.  Used  Days. 

Determine  whether  the  number  of  acute  certified  days 
were  recorded  accurately.     This  item  should  not  be 
completed  if  ACP  is 

Determine  whether  the  number  of  administrative  days 
(social  service)  were  recorded  accurately  (if  any) . 


Worksheet  Deficiency  Monitoring 


Grace 


Den.  Used  Days 


Determine  whether  the  number  of  grace  days  (if  any) 
were  recorded  accurately.     Should  be  no  more  than 
three,  Medicare  patients  only. 

Determine  whether  the  number  of  denied  days  (non- 
certified)  used  by  the  patient  were  recorded 
accurately. 


TDC 


Determine  whether  the  total  number  of  certified 
days  were  recorded  accurately. 


If  ACP  is  code  3,  this  item  should  be  blank. 


BICD 


Determine  whether  the  basis  of  assignment  of  initial 
certified  days  were  recorded  accurately.     If  ACP 
is  code  3  or  4,  this  item  should  be  blank. 


CSRs-D 


If  an  extension  was  denied,  this  item  should  be 
completed  with  a  code  2. 


Reas. 


If  any  administrative  days  were  used,  this  item 
should  be  completed. 


Tot.  Rev. 


Determine  whether  the  total  number  of  reviews 
performed  by  the  coordinator  were  recorded  accurately, 
Includes  admission  reviews,  continued  stay  reviews, 
and  interim  reviews. 


Reas. 


If  there  were  any  denied  days  used,  this  item 
should  be  completed. 


Date  Cert. 


This  item  should  be  completed  if  ACP  is  code  1,  2, 
or  4. 


Total  Def, 


Count  deficiencies  (errors)  recorded  across  in  each 
column  and  enter  the  total  under  this  item.  This 
item  will  show  the  total  number  of  errors  found  in 
each  batch. 


Totals 


This  column  should  show  the  total  number  of  errors 
found  in  each  category  for  the  month. 


Rate  :        Determine  the  error  percentage  rate  for  each 

category  for  the  month.     Overall  error  rate  for  the 
month  (total  number  of  deficiencies  divided  by  total 
number  of  worksheets  contained  in  the  batch  reviewed) 
should  be  entered  in  the  last  space  under  "Total  Def.". 

Remarks:     The  reviewer  should  note  here  any  relevant  comments  regarding 
the  findings  for  the  month. 
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CASE  STUDY 
EASTERN  CONNECTICUT  PSRO 
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Preface 


The  following  case  study  describes  the  data  system  and  data  quality 
control  procedures  developed  by  the  Eastern  Connecticut  PSRO  to  control 
the  collection  and  processing  of  the  PHDDS  and  optional  data  elements  in 
this  PSRO  area. 

We  are  indebted  to  the  Eastern  Connecticut  PSRO  for  its  support  and 
cooperation  in  the  effort  to  develop  a  data  quality  control  system  which 
will  be  useful  to  all  PSROs,  and  especially  to  Donald  Woodbury, 
Administrative  Director;  Myra  Pfisterer,  Director  of  Operations;  and 
Kathyrn  Church,  Data  Coordinator  for  their  time  and  assistance,  without 
which  preparation  of  this  case  study  would  not  have  been  possible. 
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Case  Study 

PSRO:     Eastern  Connecticut  PSRO 

PSRO  Personnel  Involved  in  Data  Activities 

Donald  Woodbury,  Administrative  Director 

Myra  Pfisterer,  Director  of  Operations 

Kathryn  Church,  Data  Coordinator 

Joseph  Burnell,  Delegated  Review  Supervisor 

MaryLyn  Sylvester,  Data  Secretary 

Jennie  Ouellette,  Key  Entry  Operator 

ENVIRONMENT 

The  Eastern  Connecticut  PSRO  area  contains  seven  hospitals  located 
throughout  the  eastern  part  of  the  State.     All  seven  hospitals  are 
delegated,  and  they  account  for  some  19,000  Federal  beneficiaries  dis- 
charged annually.      Each  hospital  completes  the  PSRO's  data  collection 
abstract  and  forwards  the  abstract  to  the  PSRO  office.     The  PSRO's  data 
processing  subcontractor  is  the  Dikewood  Corporation.     Data  entry  is 
accomplished  through  a  CRT  terminal  by  PSRO  personnel  using  Dikewood fs 
on-site  on-line  option. 

All  routine  data  and  monitoring  activities  are  described  in  detail  in 
the  PSRO's  General  Forms  Instruction  Manual.     Detailed  procedure  manuals 
are  provided  to  the  coordinators,  and  include  instructions  concerning 
performance  of  their  functions,  instructions  concerning  data  collection 
and  submission  to  the  PSRO,  and  reference  materials.     There  are  also 
detailed  procedures  concerning  the  data-related  activities  of  all  personnel 
in  the  PSRO  office.      A  sample  of  each  form  used  by  the  PSRO  is  provided. 

DATA  SYSTEM 

The  data  system  developed  by  the  PSRO  is  dependent  on  worksheets  and 
abstracts  for  input,  and  is  not  significantly  different  from  the  data 
systems  developed  by  most  PSROs.     It  is  depicted  graphically  in  Exhibit 
C-l  on  the  following  page,  and  is  described  in  the  following  narrative. 

Activities  Related  to  Data  Abstracting  and  Recording 

Gathering  of  information  for  PSRO  purposes  begins  in  the  Admissions 
Office  of  each  hospital.     All  but  one  hospital  provide  their  coordinators 
with  copies  of  the  face  sheet  for  each  Federal  beneficiary  admitted  (or 
for  each  admission  to  be  reviewed,  if  all  patient  review  is  conducted); 
the  other  hospital  provides  the  coordinator  with  an  addressograph  printout. 
The  exact  amount  of  information  received  from  the  Admissions  Office 
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*  Points  of  intervention  of  the  PSRO's  data  quality 
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varies  by  hospital,  but  generally  includes  most  of  the  UHDDS  and  the 
admitting  diagnosis.     Daily  admission  sheets  show  all  admissions  from 
the  previous  day.     These  include  anticipated  payment  source  for  each 
admission,  which  provides  a  cross-check  reference  for  the  coordinators 
concerning  all  patients  subject  to  review. 

Each  coordinator  generates  his/her  own  review  worksheet  (Exhibit  C-2) , 
which  is  completed  as  reviews  are  performed  during  the  patient's  stay. 
Each  hospital  had  an  active  PSRO  utilization  review  program,  complete 
with  forms  and  review  procedures,  prior  to  the  implementation  of  PSRO 
delegated  review,  and  in  each  hospital  the  existing  review  worksheets 
are  still  being  used.     The  PSRO  has  specified  a  minimum  set  of  informa- 
tion and  documentation  requirements  which  must  be  included  on  all 
worksheets  to  enable  the  PSRO  to  conduct  its  monitoring  activities, 
and  where  necessary,  worksheets  have  been  modified  accordingly.     At  a 
minimum,  every  worksheet  must  contain  documentation  for  every  review 
decision  made. 

Ultimately,  information  from  all  sources  is  assimilated  on  the  PSRO's 
abstract.     Each  hospital  employs  a  secretary  or  a  clerical  assistant  to 
provide  support  services  to  the  coordinator.     Among  other  functions, 
these  individuals  complete  most  of  the  abstract  for  each  Federal  patient 
discharged  with  the  exception  of  the  review  data  and  the  diagnostic  and 
procedures  codes.     Review  data  are  recorded  on  the  abstract  only  by  the 
review  coordinator.    When  the  patient  has  been  discharged  and  the 
abstract  is  complete  except  for  diagnosis  and  procedure  codes;  the 
abstract  is  sent  to  Medical  Records  for  coding. 

There  are  several  logs  which  serve  as  cross-checks  to  ensure  that  data 
are  not  lost  or  overlooked;  most  of  these  have  come  into  use  through 
the  PSRO's  initiative.     A  Statistical  Log  is  maintained  in  each  hospital 
on  a  daily  basis  for  all  Federal  patients.     Separate  logs  are  maintained 
for  Titles  V,  XVIII  and  XIX.     These  logs  contain  sufficient  information 
to  enable  completion  of  the  HSQB  121  when  the  data  are  summarized. 
In  addition,  when  abstracts  are  batched  for  submission  to  the  PSRO, 
each  individual  abstract  is  recorded  on  a  Batch  Cross-Ref erence  Log 
(Exhibit  C-3)  which  serves  to  document  exactly  which  abstracts  are 
contained  in  a  given  batch,  and  which  facilitates  recovery  of  a  document 
at  some  point  in  the  future,  if  this  becomes  necessary. 

As  a  result  of  this  supporting  documentation,  a  four-way  cross-check  is 
made  feasible  for  the  coordinators  and  for  the  PSRO's  monitoring  personnel 
the  "active"  abstracts  (whether  for  current  patients  or  discharged 
patients  for  whom  bills  must  be  certified)  can  be  checked  against  daily 
admission  sheets  for  the  appropriate  time  period,  and  also  against  the 
Statistical  Log  and  the  Batch  Cross-Ref erence  log.     This  check  of  the 
Batch  Cross-Ref erence  log  against  the  daily  discharge  sheets  enables  the 
PSRO  to  be  reasonably  sure  that  it  receives  an  abstract  for  all  Federal 
patients,  which  is  difficult  for  many  PSROs  to  verify.     At  the  time  an 
abstract  is  prepared  for  submission  to  the  PSRO,  a  tear-off  stub  is 
removed  from  each  document  and  can  be  stapled  to  the  worksheet  for  that 
patient  to  verify  that  that  abstract  has  been  completed  and  submitted. 
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ECPSRO  REVIEW  WORKSHEET 


EXHIBIT  C-2 


Ele. 
Urg. 
Emg. 


Med  tee  I  record  number 


Adm.  Date 


Disch.  Planning 


39.  Admission 
diagnosis 


Pate_ 
Hosp. 
Coor. 

60.  Admining 
physician 


Preadmit. 

certif. 


61.  Aomimen  ra»nw  NOT  performed 


o  o 


63.  ADMISSION  CERTIFICATION 


a.  Review 

No. 


b.  Date 

Month  Day 


c.  Day> 
aoproved 


d.  Reason 


CONTINUED   STAY  REVIEW 


a.  Review 


No. 

64 
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3 
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1 

b.  Dare 
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g.  Physician  Advisor  norm 


o 


f.  Coordinator  (If 
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0.  Physician  Advisor      h.  Appeal 
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Code 
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P.A 


Timeliness: 


Appropriateness; 
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Other  logs  have  been  designed  by  the  PSRO  and  are  maintained  for  specific 
purposes;  some  are  voluntarily  maintained.     An  Ancillary  Service  Review 
Log  documents  all  instances  where  ancillary  charges  have  been  reviewed 
and  justified  by  a  physician  at  the  request  of  the  fiscal  intermediary.  A 
Discharge  Planning  Log  and  a  Social  Service  Log  are  currently  completed 
voluntarily  by  more  than  half  of  the  hospitals.     These  document  activity 
and  involvement  with  the  review  process  by  the  respective  departments. 

The  PSRO  feels  that  the  quality  of  its  data  is  enhanced  significantly  by 
the  emphasis  which  has  been  placed  on  training  in  coding  and  abstracting. 
Most  of  the  hospitals  in  the  PSRO  subscribe  to  the  PAS  abstract  service, 
and  CPHA  presented  a  coding  workshop  for  the  hospitals  in  the  PSRO  area 
at  the  time  that  PSRO  review  and  data  collection  activities  began.  A 
short  time  later,  the  PSRO  sponsored  a  second  such  workshop,  which  was 
conducted  by  a  Registered  Records  Administrator.     Update  workshops  have 
been  conducted  periodically  by  PSRO  staff  personnel,  including  an  ART, 
as  specific  areas  of  difficulty  have  been  identified.     Other  training 
activity  related  to  both  coding  and  abstracting  is  conducted  on  an 
individual  hospital  basis  as  the  need  arises. 

The  hospital-employed  coordinators  have  an  advisory  group  to  the  PSRO 
which  meets  on  a  monthly  basis.     This  group  suggests  specific  areas  of 
training  which  it  feels  may  be  appropriate,  and  also  suggests  materials 
which  might  be  addressed  during  the  training  session. 

With  respect  to  diagnosis  and  procedure  coding,  the  PSRO  has  established 
specific  procedures  for  the  identification  and  sequencing  of  diagnostic 
and  procedures  codes.     The  PSRO  Board  of  Directors  determined  that  the 
attending  physician  should  determine  this  information.     A  letter  was 
sent  from  the  President  of  the  PSRO  to  the  medical  staffs  of  all  the 
hospitals  at  the  time  the  PSRO  began  its  review  activities.     This  letter 
advised  the  physicians  in  the  area  that  the  PSRO  was  required  to  develop 
profiles  on  each  individual  physician  and  that  the  PSRO  intended  to 
developed  these  profiles  from  the  information  generated  by  each  physician 
with  respect  to  his/her  own  patients.     Therefore,  principal  and  all 
secondary  diagnoses  and  procedures  are  recorded  and  sequenced  exactly  as 
indicated  in  the  medical  record  by  the  attending  physician.     If  the 
Medical  Records  Department  identifies  an  area  of  question,  it  is  encouraged 
to  contact  the'  attending  physician  for  resolution. 

Activities  Related  to  Forms  Flow  Control  and  Data  Transfer 

All  abstracts  are  received  and  coded  in  the  Medical  Records  Departments 
of  the  hospitals  following  patient  discharge.     Once  coded,  abstracts  are 
batched  for  submission  to  the  PSRO.     Batch  submission  to  the  PSRO  depends 
on  the  individual  hospital's  volume  of  Federal  patients  discharged.  The 
hospital  submits  a  batch  which  contains  either  25  or  50  abstracts  for 
both  completeness  and  logic  before  they  are  submitted  to  the  PSRO.  The 
hospitals  find  this  minimizes  error  correction  at  a  later  time  and  feel 
that  the  time  used  for  screening  is  well  spent.     Before  the  abstracts 
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are  actually  submitted,  a  Batch  Cross-Ref erence  Log,  listing  each  document 
by  medical  record  number,  is  completed.     One  copy  of  this  log  is  included 
with  the  batch  sent  to  the  PSRO;  one  copy  is  retained  in  the  coordinator's 
files.     A  batch  cover  sheet  is  attached  to  each  batch  sent  to  the  PSRO 
(Exhibit  C-A) .     Abstracts  are  mailed  to  the  PSRO  by  First  Class  Mail. 

Abstracts  for  patients  reviewed  retrospectively  are  handled  in  a  similar 
manner,  but  they  are  batched  separately  from  other  abstracts,  and  have  a 
spearate  batch  cover  sheet. 

Upon  receipt  by  the  PSRO,  counts  in  each  batch  are  verified  against  the 
information  on  the  Batch  Cover  Sheet  by  the  Data  Secretary,  who  logs  the 
batches  received  from  the  hospital  (Exhibit  C-5) .     Receipt  of  abstracts 
is  acknowledged  by  the  PSRO  immediately  in  writing.     Abstracts  are  then 
filed  by  batch,  according  to  hospital  number,  until  they  can  be  screened. 

Before  abstracts  are  entered,  they  are  pulled  by  the  Data  Coordinator 
and  screened  for  completeness  and  logic.     Screening  for  completeness 
involves  both  scanning  and  physically  touching  every  required  field  on 
the  abstract  to  insure  that  each  field  contains  an  entry.     Screening  for 
logic  involves  cross-field  checks  for  obvious  age-sex-diagnosis  inconsis- 
tencies, illogical  dates,  questionable  review  data,  etc.  Screened 
abstracts  are  refiled  to  await  key  entry. 

Errors  identified  during  this  screening  process  are  pulled.     A  Request 
for  Data  Completion  or  Correction  form  (Exhibit  C-6)  is  attached  to  the 
abstract,  which  is  logged  (Exhibit  C-7)  and  then  returned  to  the  hospital 
for  corrections.      Abstracts  containing  errors  are  corrected  and  returned 
to  the  PSRO,  and  are  checked  off  the  PSRO's  log  when  they  are  received. 

At  the  time  abstracts  are  to  be  entered  into  the  computer,  they  are 
pulled  by  batch  and  numbered  with  a  control  number  consisting  of  the 
Julian  date  and  sequential  document  numbers  for  abstracts  entered  that 
day.     The  abstracts  are  entered  by  the  key  entry  operator,  and  each 
document  is  marked  with  the  operator's  I.D.  number  when  it  has  been 
entered  and  transmitted.       The  abstracts  are  then  refiled  by  hospital 
number  and  are  retained  for  a  period  of  six  months. 

Activities  Related  to  Data  Processing  and  Output  Analysis 

Once  the  PSRO  has  transmitted  the  abstracts,  Dikewood  assumes  responsi- 
bility for  all  activities  until  the  output  reports  are  returned.  Direct 
monitoring  of  Dikewood 's  activities  is  precluded  by  geography,  but  the 
PSRO  is  satisfied  with  Dikewood' s  performance  concerning  edits,  error 
messages,  error  correction  procedures,  output  report  generation,  and 
adherence  to  confidentiality  requirements.     Any  apparent  difficulties  in 
system  operation  have  been  brought  to  the  attention  of  the  processor  and 
have  been  resolved  quickly. 
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EXHIBIT  C-4 


ECPSRO  BATCH  COVER  SHEET 

Eastern  Connecticut  PSRO 
Bridge  Street  Plaza 
41  Bridge  Street 
Willimantic,  Conn.  06226 


Hospital  Number 


Number  of  Abstracts 
in  This  Batch 


Discharge 


Month 


Year 


Batch  Number 


Cumulative  Total  of 
Abstracts  for  this  Month 


Comments : 


Signature 


Date 


ECPSRO  -  RM  Form  #  14a 
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Abstracts 

Month 
Completion 

Review 
Summary 

Ancillary 
Services 

Discharge 
Planning 

Abstracts 

Month 
Completion 

Review 
Summary 

Ancillary 
Services 

Discharge 
Planning 

 1  

Abstracts 

Month 
Completion 

Review 
Summary 

Ancillary 
Services 

Discharge 
Planning 

EXHIBIT  C-6 


Eastern  Connecticut  PSRO 
Request  for  Data  Completion  or  Correction 

Dear  , 

Please  correct  or  complete  the  attached  data  worksheet  and  return  it  with  this  form  to 
the  ECPSRO  office  as  soon  as  possible.    Thank  you. 

Hospital  _____  Month   Batch    Page  #   Chart  #   

Problem(s)  :  


HOSPITAL  COMPLETES: 

Corrected  by   (date) 

Remarks :  

ECPSRO  RM  Form  #12 


ECPSRO  COMPLETES: 
Problem  identified   19 

by   


Eastern  Connecticut  PSRO 
Request  for  Data  Completion  or  Correction 

Dear   

Please  correct  or  complete  the  attached  data  worksheet  and  return  it  with  this  form  to 
the  ECPSRO  office  as  soon  as  possible.    Thank  you. 

Hospital   Month    Batch    Page  #   Chart*   

Prob'lem(s)  :  


HOSPITAL  COMPLETES:  ECPSRO  COMPLETES: 

Corrected  by   (date)   Problem  identified   19 

Remarks :   by   


ECPSRO  RM  Form  #12 


Eastern  Connecticut  PSRO 
Request  for  Data  Completion  or  Correction 
Dear   , 

Please  correct  or  complete  the  attached  data  worksheet  and  return  it  with  this  form  to 
the  ECPSRO  office  as  soon  as  possible.    Thank  you. 

Hospital    Month   Batch    Page  #    Chart*   

Problem(s)  :   :  


HOSPITAL  COMPLETES:  ECPSRO  COMPLETES: 

Corrected  by   (date)   Problem  identified   19 

Remarks:    by  


ECPSRO  RM  Form  #12 


EXHIBIT  C-7 


LIST  OF  ABSTRACTS  REMOVED  FROM  BATCH 

Hospital  

Batch  for  _197 

Batch  

Medical  Reason 
Control  Record  Discharge  New  for 

Number  Page  Number  Date  Location  Removing 
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DATA  QUALITY  CONTROL  SYSTEM 


The  data  quality  control  system  established  by  the  Eastern  Connecticut 
PSRO  addresses  data  quality  at  each  step  from  admission  until  the  data 
are  entered  into  the  computer.     There  are  several  significant  quality 
control  activities  which  enable  the  PSRO  to  assure  itself  of  such  factors 
as : 

The  PSRO  receives  an  abstract  for  each  Federal  patient 

The  information  recorded  on  the  abstract  is  accurate,  complete,  and 
timely 

Any  abstract  can  be  located  at  any  point  in  time. 

Quality  Control  Activities  Related  to  Data  Abstracting  and  Recording 

As  discussed  previously,  the  PSRO  places  significant  emphasis  on  training 
and  has  found  that  this  emphasis  results  in  both  a  more  thorough  under- 
standing of  the  design  and  operation  of  the  data  system  and  more  effective 
collection  of  PSRO  data  at  the  hospital  level.     Routine  procedures, 
abstracting  instructions,  coding  conventions,  and  other  information  are 
included  in  the  General  Forms  Instruction  Manual  which  is  issued  to  each 
coordinator.     Periodic  procedure  memos  are  issued  as  necessary  by  the 
PSRO,  and  these  are  filed  routinely  in  the  coordinator's  manuals. 

All  abstracts  in  each  hospital  are  routed  to  Medical  Records  for  completion 
and  submission  to  the  PSRO.     The  Secretary  or  other  support  personnel  in 
each  hospital  is  responsible  for  assuring  that  an  abstract  is  initiated 
for  each  Federal  patient  admitted  as  indicated  on  the  daily  admission 
sheets.      The  coordinator  has  the  final  check  on  this  factor  and  can 
check  abstracts  against  the  daily  admission  sheets,  the  Statistical  and 
Batch-Control  logs,  and  the  bills  certified  to  verify  that  there  is  an 
abstract  for  each  Federal  patient.     The  patients  reviewed  are  checked 
off  both  the  daily  admission  and  discharge  lists.     Certification  slips 
are  attached  to  each  bill;  the  hospitals  understand  the  significance  of 
having  supporting  documentation  for  all  certifications  of  medical  necessity. 

Abstract  screening  procedures  vary  by  hospital,  but  all  hospitals  have 
someone  who  screens  abstracts  for  completeness  and  logic  before  the 
abstracts  are  submitted  to  the  PSRO.     This  screening  has  had  a  significant 
effect  on  the  number  of  abstracts  with  errors  returned  to  the  hospitals. 
The  logs  and  the  batching  and  screening  procedures  assure  that  abstracts 
are  generated  for  all  Federal  patients,  that  no  abstracts  are  lost 
within  the  hospital,  and  that  the  abstracts  are  as  accurate  as  possible 
when  they  are  submitted  to  the  PSRO. 

In  addition  to  these  procedures,  PSRO  personnel  perform  two  types  of 
monitoring  to  help  assure  the  quality  of  the  data  collected.     The  PSRO's 
concurrent  monitoring  program  is  conducted  by  the  Delegated  Review 
Supervisor,  and  combines  monitoring  of  the  review  process  in  each 
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hospital  with  review  of  data  collection  activities  within  that  hospital.  The 
PSRO's  retrospective  monitoring  program  is  conducted  by  the  Data  Coordinator 
and  addresses  the  quality  of  the  coding  and  abstracting  of  PSRO  data 
(Exhibit  C-8) . 

Concurrent  monitoring  is  performed  twice  per  month  in  each  hospital. 
There  are  specific  written  procedures  for  concurrent  review  evaluation 
which  are  to  be  followed.     The  Delegated  Review  Supervisor  visits  the 
coordinator  in  the  morning  and  works  with  the  coordinator  all  day, 
performing  exactly  the  same  work,  and  preparing  all  abstracts  and  supporting 
documentation  in  the  same  manner  as  the  coordinators.     A  number  of 
monitoring  forms  are  completed  (Exhibits  C-9  through  C-13) ,  documenting 
the  manner  in  which  the  review  activities  were  performed,  whether  deficien- 
cies observed  were  attributed  to  knowledge  or  performance,  the  manner  in 
which  Physician  Advisors  are  involved,  and  an  analysis  of  all  apparent 
deficiencies  identified. 

At  the  end  of  the  day  the  results  of  the  data  collected  are  not  compared 
with  the  coordinator's  data.     The  Delegated  Review  Supervisor  returns  to 
the  PSRO  and  subsequently  compares  the  abstracts  submitted  by  the  coordi- 
nator with  those  generated  during  the  monitoring  process.  Difficulties 
identified  have  been  documented  carefully  and  then  pursued  informally 
with  each  coordinator  on  an  individual  hospital  basis.     The  PSRO  is 
finding  that  with  the  full  implementation  of  its  data  system  and  the 
availability  of  hard  data  for  monitoring  purposes,  it  may  now  be  required 
to  document  more  formally  the  results  of  its  findings  to  someone  in 
administration  in  each  hospital.      The  PSRO  is  concerned  that  such  an 
alternative  not  jeopardize  its  established  informal  relationships  with 
the  coordinators.     Resolution  of  this  issue  is  pending,  and  its  possible 
effect  on  the  quality  of  the  data  collected  is  uncertain. 

Retrospective  monitoring  is  performed  on-site  in  each  hospital  once  per 
calendar  quarter  by  the  PSRO's  Data  Coordinator  who  is  an  A.R.T.  The 
Data  Coordinator  takes  special  monitoring  forms  to  the  hospital  and 
requests  15  records  pulled  at  random  (perhaps  15  records  whose  medical 
record  number  ends  in  "5").     She  re-abstracts  everything  on  the  PSRO's 
abstract  except  the  physician  codes  (the  hospitals  are  responsible  for 
ensuring  that  their  physicians  are  identified  accurately)  and  the  review 
data  (review  data  are  monitored  concurrently) .     The  re-abstracting  is 
checked  before  the  Data  Coordinator  leaves  the  hospital,  to  help  assure 
the  accuracy  of  the  re-abstracted  documents. 

The  Data  Coordinator  then  returns  to  the  PSRO,  pulls  the  same  abstracts 
from  its  files,  and  compares  her  re-abstracting  with  the  originals. 
Discrepancies  are  discussed  with  the  hospital  to  determine  if  an  error 
has  been  made  and  to  discuss  any  difficulties  identified.  Numerous 
coding  errors  were  found  early  in  the  re-abstracting  process,  but  such 
errors  are  decreasing  rapidly  and  now  probably  do  not  exceed  four  percent. 
The  Data  Coordinator  seeks  patterns  of  error  during  her  re-abstracting 
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EXHIBIT  C-8 


EASTERN  CONNECTICUT  PSRO 
DATA  MONITORING  -  CODING 


HOSPITAL  # 


MONITORING  DATE 


REVIEW  QUARTER 


VALIDATION  OF  MEDICAL  RECORD  #; 
VALIDATION  OF  PAYMENT  SOURCE  #: 
DIAGNOSIS: 


CODE: 


CODE  ON  FORM: 


PROCEDURE: 


CODE: 


CODE  ON  FORM: 


ECPSRO  RM  form  7e 


EXHIBIT  C-9 


EASTERN  CONNECTICUT  PSRO,  INC. 
CONCURRENT  REVIEW  ACTIVITIES  EVALUATION  REPORT 

HOSPITAL:   DATE:  

TOTAL  FEDERAL  PATIENTS:  TOTAL  REVIEWED:   


1 .    Availability  of: 

O    a-    Patient  Charts  O    c-    ECPSR0  Abstract 

O    D-    UR  Worksheets  Q    d-    Federal  Census  Report 


1.  Codes  -  Admission  Diagnosis 

2.  ECPSRO  Approved  Norms 

3.  Initial  Length-of-Stay  Assignment 

4.  Timeliness  of  Admission  Certification 

5.  Use  of  "Level  of  Care"  for  CSR 

6.  Codes  -  Additional  diagnosis, 

procedures  or  complications 

7.  Referrals  to  PA  by  UR  Coordinator 

8.  Reason  Codes 

9.  Abstracting 

NO 
PROBLEM 

PROBLEM 

KNOWLEDGE 

PERFORMANCE 

Appropriateness  of  Ancillary  Services  in  terms  of  conformance  with  pre-established 
Quality  of  Care  guidelines. 


COMMENTS: 


ECPSRO  RM  FORM  #  17 


Delegated    Review  Supervisor 


MONITORING  EVALUATION 


EXHIBIT  C-10 


Hospital  Oate  Monitor 


No. 

% 

Comments 

Pro 
No 

blem 
Yes 

Cases  Monitorpd 

Samnl  p  Tvnp'  ( irandom/hiaspH  i 

WUlll^S  IC     1  Jr            ^  1  U I  IVJ  Villi/       •  QwCU  / 

Mean  LOS 

Timely  Admission  Review 

Mean  Initial  Dav<;  fprtifipd 

Admission  Critpria  Dorumpntpd 

Vaoup  Diannosis  nn  Admission 

tu^u^    l/  i  u  m  1 1  v  j  u  «>    ui  I    n  vj  in  i  <j  J  lull 

.Reviewed  by  PA  on  Admission 

Continued  Stav  Rpuipw^ 

Total  Days  of  Stay  Extended 

Total  Administratively  Necessary  Days 

CSRs  Reviewed  by  PA 

Admission  Certifications 

Continued  Stay  Reviews 

Problem  Code: 

K  Knowledge 
P  Performance 
R  Review 
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EXHIBIT  C-ll 


MONITORING  EVALUATION 


HOSPITAL 


JAN 
No 

FEB 
No 

MAR 
No 

1 1U  • 

APR 
No 

MAY 
No 

liw  • 

JUN 
No 

JUL 
No 

AUG 
No 

SEP 
No 

OCT 
No 

NOV 
No 

DEC 
No 

Cases  Monitored 

Sample  Type  (random/biased) 

Mean  LOS 

Timely  Admission  Review 

Mean  Initial  Days  Certified 

Admission  Criteria  Documented 

Vague  Diagnosis  on  Admission 

Reviewed  by  PA  on  Admission 

Contirued  Stay  Reviews 

Total  Days  of  Stay  Extended 

Total  Administratively  Nec.Days 

CRSs  Reviewed  by  PA 

Admission  Certifications 

Continued  Stay  Reviews 

ECPSRO-Rm  Form  #  7d 
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ECPSRO  MCE  COMMITTEE 
PHYSICIAN  ADVISOR  MONITORING  FORM 


EXHIBIT  C-13 


Hospital  # 


Admission  Review 
Date  Referred 


O 


-Diagnosis  on  Admission 
Age  of  Patient   


Diagnosis  at  time  of  Referral 


Date  Monitored 


Continued  Stay  Review 
to  Physician  Advisor  # 


o 


Referral  # 


Reason  Referred: 


Complications: 


Date  of  Response   Days  Approved 

Reason  Approved:  


Documentation  Supporting  Decision 

A.  Orders  C.    Progress  Notes  E.  X-Rays 

B.  Consults  D.    Lab  Reports  F.    EKG  Reports 

G.  Other 


Reasons  Denied: 


Attending  Contacted  yes  no 

Date  of  Denial  Letter 


Date  Reviewed  by  MCE  Committee 
Committee  Decision 


Action 


ECPSRO  RM  Form  #15 


Signature  of  Chairman 


M.D. 


and  verification  activities  and  has  been  able  to  identify  cross-field 
logic  inconsistencies,  coding  from  the  alphabetic  index  alone  (rather 
than  using  the  tabular  index  also)  and  other  problems.  Follow-up 
training  programs  will  be  developed  as  required  by  the  difficulties 
identified . 

The  PSRO  is  generally  pleased  with  its  monitoring  programs,  but  has 
has  some  credibility  issues  to  resolve.     The  Delegated  Review  Supervisor 
has  four  years  of  review  (and  coding  and  abstracting)  experience,  but 
had  credibility  problems  with  Medical  Records  personnel.     The  Data 
Coordinator  is  an  A.R.T.  with  eight  years  of  coding  experience,  but 
had  some  credibility  problems  with  coordinators.     Most  of  these 
difficulties  have  now  been  resolved. 

Finally,  the  PSRO  monitors  the  quality  of  abstract  completion  through 
the  output  reports  received  from  Dikewood.     Error  reports  are  examined 
for  patterns  of  error  in  individual  hospitals  and  across  the  PSRO  area. 
Errors  which  cannot  be  resolved  at  the  PSRO  level  are  returned  to  the 
hospital  for  correction.     Reason  code  reports  are  examined  to  determine 
the  extent  to  which  coordinators  apparently  understand  and  document  the 
review  process  accurately.     The  lag  report  and  other  management  reports 
are  used  to  monitor  the  timeliness  of  data  submission. 

Quality  Control  Activities  Related  to  Forms  Control  and  Data  Transfer 

Many  of  the  quality  control  activities  in  this  area  are  among  these 
activities  described  in  the  Data  System  section.     The  quality  control 
activities  have  been  integrated  rather  thoroughly  into  the  on-going 
operation  of  the  data  system. 

Upon  receipt  by  the  PSRO,  abstracts  are  immediately  counted  by  the  Data 
Secretary  and  batch  totals  are  verified.     Any  discrepancies  are  resolved 
with  the  hospital  immediately.     Since  both  parties  have  the  Batch  Cross- 
Reference  Log  (Exhibit  C-3) ,  missing  documents  can  be  identified  quickly. 

Batches  received  are  logged  by  hospital  on  an  Incoming  Abstracts  log 
(Exhibit  C-5) ,  which  shows  the  date  and  the  number  of  documents  received. 
The  final  batch  from  a  given  month  is  headed  by  a  month  completion  form 
(Exhibit  C-14)  so  the  PSRO  knows  when  each  hospital  has  submitted  all 
abstracts  from  a  given  month,  which  also  facilitates  monitoring  of 
timeliness.      At  the  same  time,  a  card  is  sent  to  the  hospital,  notifying 
them  of  the  date,  batch  number (s)  received,  and  number  of  documents. 
This  allows  the  hospitals  to  verify  that  no  documents  are  lost  in  transit. 
Similarly,  the  PSRO  monitors  total  documents  submitted  each  month  by 
hospital  from  the  Incoming  Abstracts  Log  and  a  Batch  Control  Form  (Exhibit 
C-15)  showing  each  batch  received  from  each  hospital.     If  actual  submis- 
sions vary  significantly  from  the  expected  totals,  the  PSRO  initiates  an 
inquiry. 


C-8. 


EXHIBIT  C-14 


ECPSRO  MONTH  COMPLETION  FORM 

Eastern  Connecticut  PSRO 
Bridge  Street  Plaza 
41  Bridge  Street 
Willimantic,  Conn.  06226 


Part  1 


Completed  by  Chief,  Medical  Record  Department  or  Utilization  Review  Coordinator. 


TO 


ECPSRO 


FROM 


Hospital 


ECPSRO 

Hospital 

Number 


Month 


Year 


Completion  Date 


Day 


Month 


Year 


Total  Abstracts 


Total  number  of  batches  this  month 


Signed 


Chief,  Medical  Record  Department  or 
Utilization  Review  Coordinator 
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EXHIBIT  C-15 


ECPSRO  BATCH  CONTROL  FORM 

Eastern  Connecticut  PSRO 
Bridge  Street  Plaza 
41  Bridge  Street 
Willimantic,  Conn.  06226 


Batch 

Jan 

Feb 

Mar 

A 

5r 

[  May     j  Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Date 

No. 

Date 

No. 

Date 

No. 

Date 

No. 

8) 
+■> 

cd 
a 

No. 

Date 

No. 

Date 

No. 

Date  i 

No. 

Date 

No. 

Date 

No. 

No. 

I  Date 

No. 

1 

i 

2 

3 

4 

— i 

5 

6 

7 

8 

9 

10 
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Prior  to  data  entry  all  abstracts  are  screened  by  the  Data  Coordinator. 
Screening  procedures  involve  physically  touching  all  fields  on  the 
abstract  in  which  an  entry  is  required,  and  a  number  of  cross-field 
logic  checks.       In  instances  where  coding  is  suspect,  the  Data  Coordinator 
may  check  all  codes  in  a  given  batch  for  reasonableness  and  legitimacy 
of  codes,  consistency  between  diagnostic  and  procedure  codes,  etc.  A 
list  is  maintained  of  all  abstracts  removed  from  a  batch  for  any  reason, 
showing  both  the  reason  and  the  new  location  of  the  abstract.  Documents 
with  errors  may  be  corrected  at  the  PSRO,  but  are  usually  logged  and 
returned  to  the  hospitals.     A  Request  for  Data  Completion  or  Correction 
form  is  attached  to  the  abstract  indicating  the  specific  question  or 
error.     When  error  corrections  are  returned  by  the  hospitals  they  are 
checked  off  the  log  and  filed  for  data  entry. 

When  abstracts  are  pulled  for  data  entry,  they  are  assigned  a  control 
number  consisting  of  Julian  date  and  document  number  on  the  date  entered, 
which  facilitates  recalling  individual  documents  from  the  computer. 
Each  document  is  marked  with  the  operator's  I.D.  number  so  that  a  number 
of  abstracts  can  be  scanned  quickly  to  verify  that  they  have  all  been 
entered. 

Quality  Control  Activities  Related  to  Data  Processing  and  Output  Analysis 

This  is  the  only  area  of  the  PSRO's  quality  control  system  for  which 
there  are  no  detailed  written  procedures,  although  this  is  routinely 
monitored  by  the  PSRO.     The  data  system  developed  for  the  PSRO  was 
tested  by  Dikewood  with  50  "dummy"  abstracts  containing  known  errors.  The 
output  from  the  test  was  examined  carefully  to  assure  that  error  conditions 
had  been  identified. 

Subsequently,  the  PSRO  has  monitored  the  output  reports  for  reasonableness 
and  periodically  checks  report  totals  against  totals  from  the  PSRO's 
Incoming  Abstracts  Log  and  the  hospital's  Statistical  Logs.     Where  areas 
of  apparent  difficulty  have  been  identified,  these  have  been  discussed 
with  Dikewood,  and  problem  areas  have  been  resolved;  where  necessary, 
reports  have  been  re-run. 

Error  reports  are  received  routinely  twice  each  month,  and  other  reports 
are  received  on  a  monthly  and  quarterly  basis.     The  PSRO  knows  when 
reports  are  due  and  initiates  telephone  communication  if  reports  are 
tardy  for  any  reason. 

SUMMARY 

The  data  quality  control  system  developed  by  the  Eastern  Connecticut 
PSRO  is  very  comprehensive  and  assures  the  PSRO  of  control  over  data- 
related  activities  from  the  time  the  patient  is  admitted  until  the 
output  reports  are  received  by  the  processor.     The  quality  control 
program  places  significant  emphasis  on  training  for  all  personnel  involved 
in  data  activities,  and  all  personnel  understand  both  the  objectives  and 
the  operation  of  the  system.     The  PSRO  can  verify  that  it  receives  an 
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abstract  for  all  Federal  patients,  understands  how  coding  and  abstracting 
are  performed  in  each  hospital,  can  locate  any  abstract  at  any  point  in 
time,  and  can  monitor  the  accuracy  and  timeliness  of  all  data-related 
activities. 
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APPENDIX  D 
CASE  STUDY 

NEW  YORK  COUNTY  HEALTH  SERVICES  REVIEW  ORGANIZATION 


I 

I 

J 
I 


I 


Preface 


The  following  case  study  describes  the  data  system  and  the  data  quality 
control  program  developed  by  the  New  York  County  Health  Services  Review 
Organization  to  control  the  collection  and  processing  of  the  PHDDS  and 
optional  data  elements  in  this  PSRO  area.     In  many  respects  this  case 
will  be  particularly  useful  to  PSROs,  especially  where  a  high  volume  of 
Federal  patients  discharged  is  involved:     this  PSRO  will  eventually 
review  some  250,000  Federal  patients  discharged  each  year,  which  places 
it  among  the  largest  PSROs  in  the  nation. 

We  are  indebted  to  the  New  York  County  Health  Services  Review  Organiza- 
tion for  its  support  and  cooperation  in  this  data  quality  control  endeavor, 
and  especially  to  Eleanore  Rothenberg,  Ph.D.,  Executive  Director;  David 
Schimel,  Manager  of  the  Data  Division;  and  Elaine  Remmlinger,  Data 
Analyst  for  their  time  and  assistance,  without  which  preparation  and 
presentation  of  this  case  study  would  not  have  been  possible. 

In  addition,  special  thanks  are  due  to  these  individuals,  as  well  as 
Roger  H.  Steinhardt,  M.D.,  Medical  Director,  and  Shu-Ping  May  Lan,  Data 
Analyst  for  permitting  us  to  reference  and  describe  the  Data  Quality 
Assurance  Monitoring  Plan  which  they  have  developed.     This  plan  represents 
some  of  the  foremost  thinking  in  the  area  of  quality  control  of  PSRO 
data. 
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Elaine  Remmlinger,  Data  Analyst 
Shu-Ping  May  Lan,  Biostatistician 

5  Additional  Data  Analysts 

6  Abstractors  (in  non-delegated  hospitals) 

ENVIRONMENT 

The  New  York  County  Health  Services  Review  Organization  encompasses  36 
hospitals  throughout  Manhattan,  which  discharge  more  than  250,000  Federal 
patients  discharged  each  year.     To  date,  PSRO  review  has  been  implemented 
in  20  of  these  hospitals,  and  14  of  the  hospitals  are  delegated.  Each 
hospital  completes  the  PSRO's  data  collection  abstract,  which  is  forwarded 
directly  to  the  processor  for  data  entry.     The  PSRO's  data  processing 
contractor  is  the  Dikewood  Corporation. 

DATA  SYSTEM 

NYCHSRO's  data  system  is  noted  for  the  extent  to  which  quality  control 
functions  have  been  integrated  into  routine  system  operations. 

Activities  Related  to  Data  Abstracting  and  Recording 

The  manner  on  which  data  are  aggregated  for  recording  on  the  PSRO  abstract 
varies  widely  by  hospital  throughout  the  PSRO  area.     The  sociologic  data 
in  the  UHDDS  is  collected  primarily  from  the  face  sheet  and  the  medical 
record.     Review  data  are  transcribed  from  information  on  coordinator's 
worksheets,  and  optional  items  are  taken  both  from  worksheets  and  the 
medical  record.     In  non-delegated  hospitals,  abstracts  are  completed  by 
abstractors  who  have  Medical  Record  degrees  or  experience,  and  who  have 
been  hired  and  trained  specifically  for  this  purpose.     In  delegated 
hospitals,  either  the  coordinators  complete  the  entire  abstract,  except 
for  codes,  and  forward  the  abstract  to  the  Medical  Records  Department 
for  coding,  or  the  coordinators  forward  the  review  data  to  abstractors, 
who  complete  the  entire  document.     In  three  hospitals  a  facsimile  of  the 
PSRO's  abstract  is  generated  from  an  in-house  computer  and  transmitted 
to  abstractors,  who  complete  any  fields  for  which  the  computer  did  not 
have  information.     The  PSRO  attempts  to  gain  an  understanding,  prior  to 
delegation,  of  the  data  flow  in  each  hospital,  and  of  the  personnel 
involved  in  data  related  processes. 
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The  PSRO  provides  training  for  all  persons  performing  abstracting  functions 
this  is  required  of  PSRO  employees,  and  as  a  hospital  delegation  criterion. 
On  going  training  sessions  for  continuing  education  purposes  are  conducted 
periodically,  and  these  address  data  subjects  on  an  as-needed  basis. 
Continuing  education  can  also  be  conducted  when  NYCHSRO  staff  visits 
hospitals,  which  will  be  discussed  shortly.     NYCHSRO  does  not  conduct 
training  in  coding,  and  the  PSRO  hopes  that  personnel  employed  by  hospitals 
for  this  purpose  hold  at  least  an  A.R.T.  degree. 

The  PSRO  requires  that  information  and  review  activities  be  documented 
on  a  worksheet.     The  forms  used  vary  between  hospitals,  although  the 
PSRO  requires  sufficient  information  to  support  review,  abstracting,  and 
monitoring.     The  PSRO's  worksheet  is  used  in  all  non-delegated  hospitals. 
The  PSRO  suggests  to  its  hospitals  that  the  permanent  file  or  worksheets 
for  discharged  patients  be  maintained  separately  from  the  medical  records 
for  purposes  of  confidentiality. 

Activities  Related  to  Forms  Flow  Control  and  Data  Transfer 

Completed  abstracts  are  screened  and  batched  in  the  hospital  and  are 
collected  once  every  two  weeks.     Most  hospitals  perform  some  type  of 
abstract  screening,  although  this  varies  between  hospitals.  Abstracts 
are  screened  a  second  time  by  a  PSRO  Data  Coordinator /Analyst ,  who 
visits  each  hospital  every  two  weeks,  on  the  day  before  abstracts  are 
to  be  collected  from  that  hospital.     The  Data  Coordinator  looks  for 
omissions,  coding  errors,  and  obvious  inconsistencies.     The  next  day, 
completed  batches  are  collected  by  a  bonded  courier  and  delivered  to 
Dikewood's  New  York  Office. 

The  PSRO  requires  that  abstracts  be  completed  within  one  month  of  discharge 
Some  hospitals,  especially  large  teaching  institutions,  had  difficulties 
with  this  time  constraint  initially,  but  the  PSRO  staff  has  worked  with 
the  hospitals  on  an  individual  basis,  and  currently  only  two  hospitals 
are  encountering  significant  problems  in  this  area.     Timeliness  problems 
are  found  to  occur  primarily  because  of  late  completion  of  the  medical 
record,  or  because  of  a  backlog  of  abstracts  in  the  Medical  Records 
Department. 

When  abstracts  are  received  at  Dikewood's  office,  the  batch  totals  are 
verified,  and  abstracts  are  batched  for  data  entry.     Discrepancies  in 
batch  totals  are  communicated  to  the  PSRO,  which  contacts  the  hospital. 
The  abstract  forms  are  pre-numbered ,  and  the  batch  header  sheet  lists 
each  document  included  in  that  batch;  thus,  missing  documents  can  be 
identified  by  the  processor,  and  this  information  can  be  relayed  to  the 
hospital.     Copies  of  the  Discharge  Abstracts  Batch  Log  and  the  Discharge 
Abstracts  Transmittal  Form  are  included  as  Exhibits  D-l  and  D-2  on  the 
following  pages.     The  PSRO  serves  as  the  interface  for  all  communication 
between  the  three  organizations,  and  the  processor  does  not  have  direct 
contract  with  the  hospitals.      The  PSRO  receives  a  report  every  other 
Monday  on  the  collection  of  documents  the  preceding  Thursday  and  Friday. 
The  format  for  this  report  is  shown  in  Exhibit  D-3. 
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Hospital 


NEW  YORK  COUNTY  HEALTH  SERVICES  REVIEW  ORGANIZATION 
DISCHARGE  ABSTRACTS  BATCH-LOG 

Date 


Exhibit  D-l 


Batch  Month 
Batch  Number 


No.  of  Documents 


Signature 


Year  197 


NYCHSRO 
Document  No. 

Patient's  Last  and  First  Name 

Medical 
Record  No. 

Adm. 

Date 

Disch . 
Date 

LOS. 
Appr . 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 

21. 

22. 

23. 

24. 

25. 


NEW  YORK  COUNTY  HEALTH  SERVICES  REVIEW  ORGANIZATION 
DISCHARGE  ABSTRACTS  TRANSMITTAL  FORM 
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Hospital   

Batch-Month  Date 


Signature 


Batch  Number 

Number  of 
Documents 

NO 

Count  on 
Receipt 

T  FOR  HOSPITAL  USE 
Discrepancy 
Reconciled 

Initials 

1 

/  / 

2 

/  / 

3 

/  / 

4 

/  / 

5 

/  / 

6 

/  / 

7 

/  / 

8 

/  / 

9 

/  / 

10 

/  / 

11 

/  / 

12 

/  / 

13 

/  / 

14 

/  / 

15 

/  / 

16 

/  / 

TOTAL  BATCHES 

/  / 

ERROR  REPORTS  RETURNED                       Dates  Entered 

Pages 
Returned 

Pages 
Missing 

# 

From: 
/  / 

To: 
/  / 

# 

/  / 

/  / 

/  / 

/  / 

Received  by:   _____   Date 
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Hospital 

Pickup 
Date 

Batch  Nos. 

No.  Doc.  Rec'd. 

Date  of  Err< 
Delivered 

3r  Reports 
Returned 

Total 

Date  of  this  report   

*Due  on  Monday  following  each  pickup/delivery 


Activities  Related  to  Data  Processing  and  Output  Analysis 


Once  the  abstracts  have  been  delivered  to  Dikewood,  they  perform  all 
functions  until  the  output  reports  are  produced.     Edit  messages  encountered 
during  data  entry  are  verified  to  be  other  than  keying  errors,  and  are 
then  overriden  by  the  Dikewood  key  entry  operators,  and  these  messages 
appear  on  the  regular  error  reports.     Error  conditions  which  prohibit 
entry  into  the  file,  such  as  a  duplicate  document  number  or  an  invalid 
hospital  number,  are  relayed  immediately  to  the  PSRO,  and  are  resolved 
by  telephone  with  the  hospital.     Keyed  abstracts  are  not  returned  to  the 
PSRO  but  are  filed  in  secure  files  at  Dikewood 's  office,  by  hospital,  by 
month,  and  by  batch. 

Error  reports  are  produced  in  triplicate  every  two  weeks.     The  original 
is  sent  to  the  hospital,  and  corrections  are  made  on  the  report,  which 
is  then  returned.     Copies  of  the  report  are  retained  by  the  PSRO  and  by 
Dikewood,  so  that  they  are  aware  of  the  status  of  error  corrections,  and 
so  the  PSRO  can  identify  patterns  of  error  and  possible  areas  where 
continuing  training  efforts  may  be  required. 

DATA  QUALITY  CONTROL  SYSTEM 

The  data  quality  assurance  system  developed  by  NYCHSRO  is  designed  to 
assure  the  accuracy  of  the  data  generated  in  each  hospital  in  the  PSRO 
area.     Other  PSRO  data  quality  control  activities  provide  assurance  that 
PSRO  data  are  collected  for  each  Federal  patient  and  that  abstracts  can 
be  located  at  any  time. 

It  should  be  noted  that  NYCHSRO  has  separate  Data  and  Hospital  Review 
Departments,  and  that  these  departments  interface  with  the  hospitals 
separately;  monitoring  of  review  processes  and  activities  is  the  function 
of  the  Hospital  Review  Department,  and  it  performs  this  monitoring 
independently  of  the  Data  Division.     Where  activities  or  difficulties  in 
a  hospital  obviously  transcend  both  areas,  the  two  departments  attempt 
to  resolve  the  issue  together. 

Quality  Control  Activities  Related  to  Data  Abstracting  and  Recording 

Quality  control  related  to  the  collection  of  data  at  the  hospital  level 
begins  with  the  training  programs  described  earlier.     The  PSRO  requires 
training  of  all  personnel  performing  coding  and  abstracting  functions 
and  attempts  to  assure  that  personnel  performing  coding  functions  are 
qualified  in  that  field. 

Most  delegated  hospitals  in  the  PSRO  area  perform  all  patient  review, 
which  essentially  insures  that  all  Federal  patients  are  reviewed.  The 
admission  listings  at  all  hospitals  show  the  financial  status  and  expected 
payment  source  for  each  admission,  and  the  personnel  with  review  or 
support  responsibilities  examine  these  lists  to  verify  that  review  and 
data  collection  activities  have  been  initiated  for  each  Federal  patient. 
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This  check  is  also  performed  by  PSRO  monitoring  personnel  (from  the 
Hospital  Review  Department) ,  who  identify  a  random  sample  of  Federal 
patients  from  previous  admission  listings  and  verify  that  abstracts  are 
completed  for  these  patients.     They  also  check  to  assure  that  abstracts 
for  all  active  patients  are  in  an  active  patient  file,  that  abstracts 
for  patients  recently  discharged  are  in  a  "bill-pending"  or  similar 
file,  and  that  it  is  generally  possible  to  locate  any  abstract  at  any 
time. 

Finally,  a  Monthly  Status  Report  on  Data  Abstracting  is  prepared  by  the 
Data  Division  for  use  by  the  senior  staff.     This  report  shows  the  current 
status  of  abstracting  activities  at  each  hospital,  including  any  accumulated 
backlog  and  need  for  assistance  from  the  PSRO.     Where  problems  exist, 
the  reason  for  the  problem,  breakdown  of  the  backlog  by  month  of  discharge, 
and  payment  status  is  provided.     Recommended  actions,  and  the  projected 
situation  and  actions  during  the  coming  month  are  included. 

Emphasis  is  placed  on  assuring  the  quality  of  abstracted  data  before  the 
abstracts  leave  the  hospital.     Most  hospitals  have  a  screening  process 
for  review  of  abstracts  in  order  to  minimize  the  effort  which  results 
later  when  the  error  reports  are  produced.     In  addition,  every  two  weeks 
a  PSRO  Data  Coordinator /Analyst  visits  each  hospital  on  the  day  before 
abstracts  are  collected  by  the  Dikewood  courier.      The  Data  Coordinator/ 
Analyst  reviews  all  abstracts  to  be  submitted,  looking  for  omissions, 
coding  errors,  and  any  obvious  inconsistencies.       In  addition,  NYCHSRO 
has  developed  a  sophisticated  retrospective  re-abstracting  plan  designed 
to  yield  statistically  significant  conclusions  about  data  quality.  The 
following  discussion,  while  brief,  summarizes  the  key  aspects  of  this 
plan.     Because  of  space  limitations,  none  of  the  statistical  calculations 
through  which  the  parameters  and  standards  were  determined  have  been 
included. 

Data  Quality  Assurance  Monitoring  Plan 

The  data  quality  assurance  monitoring  plan  developed  by  NYCHSRO  provides 
for  quarterly  retrospective  monitoring  of  PSRO  data  to  identify  errors 
committed  at  the  hospital,  by  the  data  abstractor,  or  at  the  data  entry 
facility  by  the  data  processor.     A  statistical  sampling  methodology  was 
developed  for  item  selection  which  is  expected  to  yield  assurances  that 
the  actual  error  rate  in  the  PSRO's  data  is  not  greater  than  one  percent. 

The  actual  data  verification  process  concerning  activities  in  the  hospital 
involves  visual  screening  in  the  hospital,  by  a  NYCHSRO  Data  Coordinator/ 
Analyst,  of  the  original  abstract  submitted  by  the  hospital  against  the 
original  source  documents  (the  medical  record  and  the  coordinator's 
worksheet) .     This  is  not  the  bi-weekly  screening  conducted  before  documents 
are  transmitted  to  Dikewood.     The  only  exception  to  this  procedure 
occurs  for  diagnostic  and  procedure  codes,  which  are  re-coded  by  the 
Data  Analyst  from  the  medical  record  and  then  checked  against  the  infor- 
mation reported  on  the  abstract.     A  computer  print-out  of  the  documents 
to  be  reviewed  is  obtained  from  Dikewood,  and  comparison  of  this  report, 
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the  abstract,  and  the  source  documents  reveals  which  errors  occurred 
during  the  data  entry  process.     Monitoring  forms  are  used  to  record  all 
errors  identified,  and  will  ultimately  be  used  to  calculate  the  quality 
of  the  data  from  each  hospital. 

Analysis  of  the  findings  of  this  process  is  facilitated  by  classifi- 
cation of  types  of  errors  and  by  the  definition  of  levels  of  significance 
for  data  items.     Errors  are  classified  as  omission  of  transcriptional. 
Diagnostic  and  procedure  coding  errors  are  classified  as  "marginal"  or 
"significant."    Data  items  are  categorized  into  three  levels  of  significance, 
depending  on  the  effect  that  an  error  in  the  given  element  would  have  on 
the  reliability  and  validity  of  the  profiles  and  norms  to  be  developed. 

The  PSRO  requires  95  percent  confidence  level  concerning  the  acceptance 
of  hospitals  whose  error  rates  do,  in  fact,  fall  below  one  percent. 
Different  confidence  limits  were  examined  with  respect  to  sample  sizes 
for  monitoring,  and  sample  sizes  will  be  selected  by  volume  of  Federal 
beneficiaries  discharged,  as  follows: 

-  In  hospitals  with  1,500  more  discharges  during  a  three-month  period, 
100  cases  are  to  be  reviewed 

-  In  hospitals  with  750-1,499  discharges  during  a  three-month  period, 
75  cases  are  to  be  reviewed 

In  hospitals  with  fewer  than  750  discharges  during  a  three-month 
period,  50  cases  are  to  be  reviewed. 

Acceptance  or  rejection  of  the  quality  of  the  data  is  based  on  the 
number  of  documents  sampled  which  contain  errors.     Options  of  sample 
size  relating  to  the  desired  confidence  limits  were  examined,  and 
acceptance  numbers  (the  maximum  number  of  errors  in  a  sample  with  which 
the  quality  of  the  data  from  that  hospital  will  still  be  accepted)  and 
rejection  numbers  (the  minimum  number  of  errors  for  which  the  quality  of 
the  data  from  that  hospital  will  be  rejected)  for  each  category  of 
sample  sizes  were  identified.     The  quality  control  program  is  to  be 
conducted  in  the  manner  described  above.      Where  problem  areas  are 
identified,  the  PSRO  will  implement  remedial  training  activities. 

Quality  Control  Activities  Related  to  Forms  Flow  Control  and  Data  Transfer 

Quality  control  concerning  the  transfer  of  PSRO  data  begins  in  the 
hospital  with  the  Discharge  Abstract  Batch  Log  (Exhibit  D-l) ,  which  is 
a  cumulative  list  of  all  abstracts  within  a  given  batch  becomes  the 
batch-header  sheet  for  that  batch.     A  Discharge  Abstracts  Transmittal 
Form  (Exhibit  D-2)  lists  all  batches  collected  from  a  given  hospital  at 
one  time,  and  is  signed  by  the  courier  who  collects  the  documents. 
Copies  of  both  forms  are  retained  in  the  hospitals.     Dikewood  checks 
all  batch  totals  on  receipt  and  verifies  the  number  of  documents  received. 
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When  abstracts  are  received  at  Dikewood,  the  totals  are  verified,  abstracts 
are  batched  for  data  entry,  and  a  NYCHSRO  Abstract  Control  Log  Sheet 
(Exhibit  D-4)   is  attached  to  the  batch  folder.     The  number  of  batches 
received  and  the  total  number  of  abstracts  included  are  entered  on  a 
continuous  log,  the  Hospital  Summary  Log  Sheet,  by  hospital  (Exhibit  D- 
5) .     Any  discrepancies  between  batch  totals  or  abstract  totals  are 
referred  immediately  to  the  PSRO,  which  resolves  the  problem  with  the 
hospital. 

Quality  Control  Activities  Related  to  Data  Processing  and  Output  Analysis 

When  completed  batches  are  ready  for  keying,  the  batch  number  is  entered 
and  re-entered  to  insure  accuracy,  and  the  abstracts  are  entered.  At 
the  end  of  the  batch,  the  total  number  of  documents  shown  on  the  Abstract 
Control  Log  is  verified  against  the  total  shown  on  the  terminal,  to 
insure  that  none  were  omitted.     Any  error  messages  which  appear  on  the 
terminal  at  this  point  are  overriden  by  the  key  entry  operator,  unless 
they  prevent  access  to  any  file  (such  as  a  duplicate  document  number  or 
an  invalid  hospital  number) ,  in  which  case  the  PSRO  is  contacted  immediately 
and  resolves  the  question  with  the  hospital. 

Error  reports  resulting  from  this  point  of  entry  are  usually  generated 
within  two  weeks  of  the  time  of  input.     Error  reports  are  printed  from 
the  printer  in  Dikewood' s  New  York  Office.     They  are  sent  to  the  PSRO 
for  approval,  and  are  then  transmitted  directly  to  the  hospitals  for 
corrections.     Corrections  are  made  on  the  error  reports,  and  are  returned 
to  the  PSRO  for  re-submittal  to  Dikewood. 

On  an  on-going  basis,  the  PSRO  monitors  the  timeliness  and  validity  of 
reporting  requirements,  as  well  as  the  accuracy  of  the  data  entry 
operation.     Any  errors  or  difficulties  with  deliverables  are  brought  to 
the  attention  of  the  processor,  which  makes  the  appropriate  corrections. 

SUMMARY 

The  data  quality  control  system  developed  by  the  New  York  County  Health 
Services  Review  Organization  apparently  addresses  the  data  control  needs 
of  this  high  volume  PSRO  quite  well.     The  most  significant  data  problems 
encountered  by  the  PSRO  on  an  on-going  basis  is  timeliness  and  backlogs 
of  abstracts  in  some  hospitals.     The  PSRO  is  able  to  track  abstracts 
throughout  the  data  system,  and  can  be  reasonably  sure  that  it  receives 
an  abstract  for  all  Federal  patients  in  a  given  hospital.     Of  special 
note  is  the  data  quality  assurance  monitoring  plan,  which  will  provide 
the  PSRO  with  highly  reliable  assurances  that  the  data  accuracy  rate  in 
any  given  hospital  does  or  does  not  fall  within  the  limits  of  acceptable 
performance  established  by  the  PSRO. 
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NYCHSRO  ABSTRACT  CONTROL  LOG  SHEET 
(Attach  to  Batch  Folder) 
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SOUTH  CAROLINA  MEDICAL  CARE  FOUNDATION 
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Preface 


The  following  case  study  describes  the  data  system  and  data  quality 
control  procedures  developed  by  the  South  Carolina  Medical  Care  Founda- 
tion to  control  the  collection  and  processing  of  the  PHDDS  and  optional 
data  elements  in  the  PSRO  area. 

We  are  indebted  to  the  South  Carolina  Medical  Care  Foundation  for  its 
support  and  cooperation  in  the  effort  to  develop  a  data  quality  control 
system  which  will  be  useful  to  all  PSROs.     In  particular,  we  wish  to 
thank  William  F.  Mahon,  Executive  Director  and  Carolyn  S.  Lipp,  PSRO 
Data  Manager,  without  whose  assistance  preparation  of  this  case  study 
would  not  have  been  possible. 
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CASE  STUDY:     SOUTH  CAROLINA  MEDICAL  CARE  FOUNDATION 


QUALITY  CONTROL  PERSONNEL 

Carolyn  Lipp,  A.R.T.  —  Data  Manager 
Malinda  Anderson    -  Data  Clerk 

Manual  screening  of  all  abstracts 
Tony  Stokes,  R.R.A.  —  Program  Surveyor 

Validation  Survey  Team  member 
Sylvia  Harrison,  R.N.  —  Program  Surveyor 

Validation  Survey  Team  member 

ENVIRONMENT 

The  South  Carolina  Medical  Care  Foundation  is  a  statewide  PSRO  that 
currently  performs  review  on  Federal  patients  only  except  for  a  pilot 
project  in  three  hospitals  where  Blue  Cross  patients  are  reviewed. 
There  are  74  hospitals  in  the  State  of  which  69  are  under  review  (the 
other  five  do  not  admit  Federal  patients).     Of  the  69  hospitals,  33  are 
delegated  for  concurrent  review.     The  PSRO  abstract  is  used  in  all  of 
the  hospitals  under  review;  the  activities  of  data  collectors  are  not 
integrated  into  the  PSRO's  system  (about  70  percent  of  the  discharges  in 
the  State  are  covered  by  PAS).     South  Carolina  has  about  130,000  Federal 
patients  discharged  per  year.     The  PSRO's  data  processor  is  Blue  Cross/ 
Blue  Shield  of  South  Carolina. 

The  most  recent  data  tape  (January  —  March  1977)  submitted  to  HSQB  was 
reported  by  the  Federal  processor  to  have  a  0.0  percent  error  rate 
(using  PSRO  Transmittal  No.  28  edit  checks)  based  upon  33,700  records. 

GENERAL  DATA  SYSTEM 

Data  Abstracting  and  Recording 

In  all  hospitals,  the  admitting  office  sends  a  daily  list  of  the  previous 
day's  admission  to  the  Review  Coordinator.     This  daily  list  includes  the 
patient  name,  room  number,  attending  physician's  name,  pay  source  and 
(usually)  the  medical  record  number.     The  list  does  not  have  a  standard 
format.     The  Review  Coordinator  locates  the  medical  record  on  the  floor 
and  records  social  and  demographic  information  onto  the  abstract.  The 
Review  Coordinator  verifies  the  surgery  schedule  with  the  medical  record 
and  codes  the  admitting  diagnosis  and  procedure  onto  the  abstract. 
Performance  of  admission  review  is  in  accordance  with  a  manual  developed 
by  the  PSRO  and  used  by  all  hospitals.     The  Review  Coordinator  is  required 
to  maintain  a  worksheet  thoroughly  documenting  the  review  process.  This 
support  includes  documentation  supporting  ancillary  charges  and  Physician 
Advisor  activity  (the  PA  must  record  his/her  activities  on  the  worksheet) . 
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The  Review  Coordinator  files  the  worksheet/abstract  form  set  in  a  tickler 
file.     Continued  stay  review  is  instigated  by  the  file  and  performed  as 
described  above. 

Upon  patient  discharge,  the  medical  record  is  routed  to  the  hospital 
Medical  Records  Department.     There,  the  diagnoses  and  procedures  on  the 
face  sheet  are  coded  and  a  copy  is  sent  to  the  Review  Coordinator. 

The  Review  Coordinator  receives  a  daily  discharge  listing.     This  is  used 
to  assure  that  all  Federal  patients  have  undergone  complete  review.  The 
daily  discharge  listings  are  maintained  in  a  notebook.  Additionally, 
the  Review  Coordinator  must  certify  the  bill  for  payment.     Since  local 
PSRO  policy  requires  that  the  worksheet,  abstract,  and  bill  be  simul- 
taneously examined  for  comparability,  the  abstract  cannot  be  submitted 
until  the  bill  is  prepared  and  certified. 

Retrospective  review  is  performed  in  essentially  the  same  fashion  as 
concurrent  review:     the  chart  is  retrieved  and  reviewed  as  if  the 
patient  were  still  in  the  hospital.     The  Review  Coordinator  still  makes 
level  of  care  determinations  and  still  must  certify  the  bill  for  payment. 
The  only  difference  is  that  the  PSRO  cannot  terminate  benefits  (make 
retrospective  denials) .     The  Review  Coordinator  assigns  Acute  Certified 
Days  only  as  those  deemed  medically  necessary  by  the  standard  review 
process,  not  necessarily  the  actual  length  of  stay  that  appears  on  the 
chart.     For  Medicare  and  Medicaid  patients,  differences  between  the  two 
are  paid  as  administratively  necessary  days. 

Forms  Flow  Control  and  Data  Transfer 

Abstracts  are  mailed  from  the  hospital  to  the  PSRO  every  Friday.  The 
Review  Coordinator  keeps  the  worksheet;  these  are  filed  alphabetically 
by  patient  name  for  two  years.     There  is  no  second  copy  of  the  abstract; 
the  coordinator  worksheet  is  the  second  copy  of  the  data  set.     If  the 
coordinator  wishes  to  retain  a  copy  of  the  abstract,  she  makes  a  photocopy; 
however,  most  coordinators  have  found  no  need  to  retain  a  copy. 

In  small  non-delegated  hospitals  where  a  full  time  coordinator  is  not 
required,  the  PSRO  reimburses  that  facility  on  a  per-abstract  basis 
utilizing  a  formula  whereby  the  coordinator's  existing  salary,  fringes, 
space,  supplies,  postage,  and  telephone  are  considered  in  establishing 
the  rate  per  abstract.     This  rate  will  vary  from  facility  to  facility. 
Batch  control  slips  (Exhibit  E-l)  accompany  batches  only  from  those 
hospitals  receiving  reimbursement  for  review.     These  slips  are  validated 
by  the  Data  Clerk  and  forwarded  to  the  PSRO  Controller  for  payment  to 
the  hospital. 

The  PSRO  maintains  a  weekly  log,  by  hospital,  indicating  the  number  of 
abstracts  submitted,  the  number  with  errors,  the  number  of  error  documents 
recieved,  the  number  of  error  documents  with  errors  and  the  weekly  error 
percentage.     This  weekly  log  is  generated  by  the  data  processor.  The 
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Exhibit  E-l 


P  S  R  0 

ABSTRACT  CONTROL  SHEET 

For  period  from   to 

Total  number  of  Abstracts  in  this  batch   

Co-ordinators  signature:   


hospitals  are  required  to  submit  their  total  number  of  Federal  beneficiaries 
discharged  to  the  PSRO  on  a  quarterly  basis  (Exhibit  E-2) .     This  allows 
the  PSRO  to  compare  the  number  of  discharges  to  the  number  of  abstracts 
received.     Discrepancies  are  then  forwarded  to  the  PSRO  Lead  Coordinator 
for  investigation.      Additionally,  this  comparison  is  utilized  by  the 
Validation  Survey  Team  (see  below)  on  a  semi-annual  basis  when  monitoring 
visits  occur  at  the  hospital.     Since  the  hospitals  maintain  the  daily 
discharge  listings  in  a  notebook,  discrepancies  between  the  number 
submitted  by  the  hospital  and  the  number  counted  by  the  processor  can  be 
validated.     If  a  discrepancy  is  detected,  the  hospital  can  request 
(through  the  PSRO)  a  complete  listing  of  the  discharge  records  received 
and  counted  by  the  processor  for  the  period  in  question.      The  hospital 
can  then  note  what  names  are  missing  from  the  listing  and  take  steps  to 
(re)  submit  those  abstracts.     To  date,  only  two  major  discrepancies  have 
been  discovered  and  both  were  resolved  through  joint  efforts  of  the  PSRO 
and  the  hospital. 

Between  the  PSRO  and  data  processor,  abstracts  are  batched  in  groups  of 
100;  the  100  abstracts  are  in  hospital  order  but  more  than  one  hospital's 
abstracts  may  be  in  an  individual  batch  if  the  hospital  has  submitted 
more  or  less  than  100  abstracts  for  that  week.     Each  abstract  is  assigned 
a  unique  identification  number  composed  of  the  Julian  date  and  a  sequential 
number.     The  batch  I.D.  number  is  logged  at  the  PSRO.     The  processor's 
computer  lists  the  record  numbers  contained  under  each  batch  I.D.  number. 
If  batch  counts  at  the  processor  are  less  than  100,  the  batch  control 
report  from  the  processor  will  indicate  the  batch  is  out  of  balance. 
The  Data  Clerk  reviews  the  batch  control  report  when  the  processed 
abstracts  are  returned  and  ascertains  which  abstracts  are  missing  or  are 
numbered  incorrectly.      Any  problems  are  corected  by  the  Data  Clerk. 

Edit  messages  and  corrections  are  not  batched  from  the  processor  to  the 
PSRO.     The  PSRO  returns  corrections  to  the  processor  in  batches  of  50. 

Data  Entry  and  Edit 

The  PSRO  timeliness  policy  is  that  all  abstracts  must  be  submitted 
within  30  days  after  discharge.     In  practice,  most  hospitals  submit  data 
30-60  days  after  discharge;  the  PSRO  estimates  that  70-75  percent  of 
abstracts  are  received  within  60  days  after  discharge.      The  primary 
reason  for  tardiness  appears  to  be  slowness  in  completion  of  the  medical 
record  by  the  attending  physician;  this  accounts  for  approximately  60 
percent  of  the  late  records.     Another  30  percent  of  late  abstracts  is 
caused  by  Billing  Office  slowness  in  bill  preparation  (local  PSRO 
policy  requires  that  the  abstract  cannot  be  submitted  until  the  bill  is 
certified) .     The  last  ten  percent  is  time  lag  in  applications  to  the 
intermediaries  for  coverage  of  patients. 
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Exhibit  E-2 


QUARTERLY  DISCHARGE  STATISTICS 
TO  PSRO 


HOSPITAL   .   DATE   July  12,  1977 

QUARTER   April  1,  1977    TO  June  30,  1977 

TOTAL  MEDICARE  DISCHARGES  117 


TOTAL  MEDICAID  DISCHARGES   38     +     9NB     =  47 

TOTAL  TITLE  V  DISCHARGES  0 


Signed 


Position 


Please  complete  and  submit  within  15  days  following  the  end  of  each  quarter  to: 


Carolyn  Lipp,  A.R.T. 
PSRO  Data  Manager 

South  Carolina  Medical  Care  Foundation 
P.  0.  Box  21667 

Columbia,  South  Carolina  29221 


All  abstracts  are  screened  at  the  PSRO  by  the  Data  Clerk.     (See  Data 
Quality  Control  Procedures  for  description) .     If  errors  are  detected  at 
this  point,  the  abstract  is  logged  and  returned  to  the  hospital  (Exhibit 
E-3) .     If  no  errors  are  detected,  the  abstracts  are  batched  and  submitted 
to  the  processor  on  a  weekly  basis.     The  abstracts  are  keyed  and  subjected 
to  the  processor's  edits.     Errors  are  recorded  on  a  computer-generated 
error  document.     The  error  documents  are  returned  to  the  PSRO  weekly. 
The  PSRO  files  and  the  abstract;  the  error  document  is  logged  out  and 
mailed  to  the  Review  Coordinator.     The  Review  Coordinator  makes  the 
correction  directly  on  the  error  document  and  returns  it  to  the  PSRO, 
where  it  is  logged  in.     The  PSRO  resubmits  the  error  document  to  the 
processor.     The  processor  enters  and  edits  the  error  document  in  conjunc- 
tion with  the  keyed  abstract  (which  was  held  in  a  suspense  file) .  If 
the  corrected  abstract  passes  the  edits,  it  is  merged  with  the  data 
base.     If  not,  the  correction  process  is  repeated. 

Data  Processing  and  Output  Analysis 

Output  reports  are  evaluated  by  various  management  level  employees  in 
the  PSRO  office  (i.e.,  Medical  Director  evaluates  all  reports  pertaining 
to  attending  physician  and  physician  advisor  performance;  Operations 
Manager  evaluates  all  reports  pertaining  to  Coordinator  performance; 
Data  Manager  evalutes  all  reports  pertaining  to  Coordinator  performance; 
Data  Manager  evaluates  lag  analysis,  error  rates,  length  of  stay, 
rankings,  and  other  reports  of  that  nature;  the  Executive  Director 
screens  all  reports  but  pays  close  attention  to  the  reports  by  pay 
source  that  are  shared  with  the  intermediaries) . 

Additionally,  the  Data  Manager  monitors  all  reports  for  possible  varia- 
tions generated  by  inaccurate  data  or  processor  programming  errors.  A 
report  request  form  is  completed  on  a  monthly  basis  by  the  Data  Manager 
and  submitted  to  the  Processor.     This  request  indicates  which  reports 
are  being  requested  and  for  what  period.      A  copy  of  this  request  is 
retained  and  when  reports  are  received  from  the  processor,  all  reports 
are  accounted  for  by  comparing  the  reports  received  to  what  was  requested. 

DATA  QUALITY  CONTROL  PROCEDURES 

Data  Abstracting  and  Recording 

1.  Training 

All  Review  Coordinators  from  all  hospitals  attend  an  initial  training 
session  and  periodic  sessions  thereafter.     New  Review  Coordinators 
attend  a  one-week  session  that  covers: 

.     PSRO  Law  (P.L.  92-603) 

Organization  of  the  South  Carolina  Medical  Care  Foundation 
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ABSTRACT  ERROR  SHEET 


HOSPITAL 


DATE  

Itmes  to  be  completed  for  corrected. 


Please  return  corrected  abstract  with  error  sheet  attached  to  the  South 
Carolina  Medical  Care  Foundation  as  soon  as  possible. 


Coverage  issues 


Coding 

Abstract  completion 

Relationship  edits 

Error  correction  procedures 

Concurrent  review 

Confidentiality 

Course  elements  of  coding,  abstract  completion,  relationship  edits, 
confidentiality  and  error  correction  procedures  are  conducted  by 
the  PSRO  Data  Manager,  who  is  an  Accredited  Record  Technician. 
Other  elements  are  taught  by  Lead  Coordinators. 

Every  three  or  four  months,  the  PSRO  holds  advanced,  on-going 
training  sessions  (Exhibit  E-4) .     Attendance  from  hospitals  is  not 
mandatory  but  is  strongly  urged  by  the  PSRO.     Delegated  hospitals 
may  choose  to  send  only  one  coordinator  or  all  of  them.     In  most 
cases,  all  the  coordinators  attend.     PSRO-employed  coordinators  are 
required  to  attend.     Advanced  sessions  cover: 

Unusual  abstracting  procedures 

Abstracting  and  coding  retraining 

New  procedures 

Discussions  of  error  rates  and  lag  analysis 

Questions  and  answers. 

To  supplement  the  training  activities,  procedure  bulletins  (Exhibit 
E-5)  are  sent  out  to  cover  special  or  changing  procedures.  The 
Review  Coordinator  is  required  to  maintain  these  in  a  notebook. 

The  Review  Coordinator  is  responsible  for  performing  Admit  Diagnosis 

and  Procedure  coding.     In  South  Carolina,  approximately  one-half  of 

the  coordinators  are  nurses  and  the  other  half  are  A.R.T.'s  or 
R.R.A. 's. 

2 .      Ongoing  Activities 

The  Review  Coordinator  applies  visual  edits  to  each  abstract  before 
it  is  submitted  to  the  PSRO.     These  edits  are  those  recorded  in  the 
PSRO  Data  Manual.     Each  abstract  is  screened  for  accuracy  (the 
edits)  and  completeness.     The  Coordinator  may  occasionally  verify 
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EXHIBIT  E-4 


SOUTH  CAROLINA  MEDICAL  CARE  FOUNDATION 

P.  O.  BOX  11188.        3325  MEDICAL  PARK  RD..       COLUMBIA.  S.  C.  29211        (803)  77&-4780 


AGENDA 


COORDINATORS  WORKSHOP 
April  7,  1976 
Ramada  Inn 


9:30     -     9:45  OPENING  REMARKS 

Dr.  Kenneth  N.  Owens,  President 

South  Carolina  Medical  Care  Foundation 

9:45     -     9:50  INTRODUCTION  OF  GUESTS  AND  STAFF 

Mr.  William  F.  Mahon 
Executive  Director  -  PSRO 

9:50     -     10:15  CONDUCT  OF  RECONSIDERATIONS 

Mr.  Hal  M.  Stone,  Jr. 
Professional  Relations  Manager  -  PSRO 

10:15  -     10:30  BREAK 

10:30  -     11:00  CERTIFICATION/RECERTIFICATION 

Mrs.  Doris  Covert,  R.N. 
Operations  Manager  -  PSRO 

11:00  -     11:45  ERROR  CORRECTION  REQUEST  FORM 

Ms.  Carolyn  Lipp,  ART 
Data  Manager  -  PSRO 

11:45  -    2:00  LUNCH  " 

1:00    -     1:30  GUIDELINES  FOR  HEALTH  CARE  EVALUATION 

Ms.  Carolyn  Lipp,  ART 
Data  Manager  -  PSRO 

1:30     -     2:00  HEALTH  CARE  SERVICES  REVIEW 

Mrs.  Oteria  S.  Marsden,  R.N. 

PSRO  Lead  Program  Review  Coordinator 

2:00     -     3:00  REVISED  RS  9-74 

Ms.  Carolyn  Lipp,  ART 
Data  Manager  -  PSRO 

3:00     -     3:15  BREAK 

3:15     -     4:00  REVIEW  AND  DISCUSSION 


PROFESSIONAL     STANDARDS     REVIEW  ORGANIZATION 


EXHIBIT  E-5 


rocedure  Bulletin  No.  29 


February  9 ,  1977 


TO: 


ALL  COORDINATORS 


FROM: 


Carolyn  Lipp 
PSRO  Data  Manage. 


SUBJECT : 


TITLE  V  ABSTRACTING  INSTRUCTIONS 


Please  abstract  Title  V  patients  as  all  others  except  for  the 
following : 

1.  Field -Patient  ID  No.  -  This  number  will  be  obtained  from  the 
Authorization  Sheet.     Be  sure  to  place  a  bar  over  the  letter. 

2.  Field  #5  -  Pay  Source  -  Please  mark  pay  source  5  (Gov't). 
If  Medicaid  is  paying  part  of  the  bill,  then  Medicaid  is 
primary  and  should  be  the  only  pay  source  marked. 

3.  Field  £25  -  LOS  Criteria  -  Indicate  a  code  4  for  a  crippled 
child  and  a  code  5  for  a  maternity  patient. 

4.  Field    £32  -  Sex  -  You  will  occasionally  have  a  child  that 
the  sex  has  not  been  determined  until  surgery.     Please  delay 
marking  the  sex  until  after  surgery  and  then  indicate  appro- 
priate sex.     Remember  -  surgical  procedures  and  diagnoses  that 
are  sex  specific  must  be  compatible  or  you  will  get  an  error 
document.     Watch  these  patients  closely  as  you  may  get  a 
hysterectomy  on  a  male  patient  or  a  prostatectomy  on  a  female. 

5.  Field  £34  -  Admission  Source  -  Outpatient  clinic  here  does 

not  necessarily  refer  only  to  the  hospital's  outpatient  clinic. 
You  will  be  getting  Title  V  patients  directly  from  their  out- 
patient clinics  and  these  should  be  so  indicated. 

6.  Field  #52  -  Adm.  Nec.  Cert.  Days  -  Indicate  all  days  in  this 
space  that  are  not  acute  certified  days.     Remember  -  no  termina 
tions  on  these  patients. 

7.  Field  £54  -  Adm.  Nec.  Reason  -  Please  add  reason  code  #44  to 
your  list  -  "not  medically  necessary  -  Title  V".     Only  use  this 
reason  code  for  Title  V  patients. 


If  you  have  any  questions,  please  let  me  know. 


codes  recorded  by  the  Medical  Records  Department.     An  additional 
quality  check  performed  by  the  Review  Coordinator  is  the  comparison 
of  the  abstract,  work-sheet,  and  bill  before  the  bill  is  certified 
for  payment. 

PSRO  Lead  Coordinators  periodically  review  random  samples  of 
documents  for  quality  control  purposes.     Such  monitoring  includes 
completeness  and  checking  for  accuracy  through  comparison  with  the 
abstract,  medical  record,  and  worksheet.      The  Data  Manager  refers 
the  volume  and  type  of  errors  to  the  Lead  Coordinators  as  necessary. 
Technical  assistance  is  then  provided  to  that  coordinator  by  the 
Lead  Coordinator  for  that  region. 

The  PSRO  maintains  two  WATS  lines.     These  are  for  use  by  Coordinators 
when  review/abstracting  problems  are  encountered.  Abstracting 
problems  have  occasionally  included  diagnoses  that  do  not  appear  in 
the  code  books.     In  these  situations,  the  PSRO  has  requested  code 
assignments  from  the  American  Medical  Records  Association. 

The  PSRO  maintains  a  permanent  "Validation  Survey  Team"  composed  of 
a  Registered  Record  Administrator  and  a  Registered  Nurse.  The 
Team  visits  delegated  hospital  semi-annually  and  non-delegated 
hospitals  annually  to  evaluate  the  PSRO  program  functioning  in  each 
institution.     In  terms  of  data  activities,  the  Team  does  not  re- 
abstract  but  rather  validates  abstracts  through  comparison  with  the 
medical  record  and  worksheet.     The  abstract  sample  is  pulled  on  a 
random  basis  from  the  PSRO's  files  of  abstracts.       Sample  size  is 
approximately  10-36  discharge  records  and  is  based  on  the  total 
number  of  Federal  beneficiaries.     Additionally,  an  equal  number  of 
inpatients  are  reviewed  by  the  Team.     During  the  visit  to  the 
hospital,  the  appropriate  medical  records  and  worksheets  are  pulled 
and  examined.     Every  field  on  the  abstract  is  examined:  the 
R.R.A.  reviews  the  data  portions  of  the  abstract  while  the  R.N. 
scrutinizes  the  level  of  care  and  review  process  items. 

In  preparation  for  each  visit,  the  Team  reviews  error  percentage 
reports  and  lag  analysis  reports  from  the  processor.     Upon  comple- 
tion of  the  visit,  the  Team  holds  a  single  exit  interview  with  bozh 
the  hospital  administrator  and  (ocasionally)  the  Review  Coordinator (s) . 
A  written  report  letter  is  prepared  describing  the  visits;  copies 
are  submitted  to  the  hospital  administrator,  Chief  of  Staff,  Review 
Coordinator  and  the  PSRO  Peer  Review  Committee.     The  Peer  Review 
Committee  reviews  the  letter  and  takes  appropriate  action.  This 
action  has  so  far  consisted  of  resurveys  after  a  period  of  time 
during  which  the  hospital  is  to  correct  any  deficiencies  detected 
by  the  Validation  Survey  Team. 
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Adequate  data  quality  is  a  condition  of  delegation.     If  data  are 
inaccurate  and/or  not  timely,  delegation  may  be  removed.      To  date, 
no  such  action  has  occurred.     One  hospital  received  verbal  and 
written  warnings  that  its  status  was  in  jeopardy.      In  its  efforts 
to  remain  delegated,  the  hospital  developed  one  of  the  finest 
quality  control  systems  in  the  State. 

Forms  Flow  Control  and  Data  Transfer 

The  on-site  Validation  Survey  Team  activities  include  monitoring  of  the 
integrity  of  abstract  transfer  from  the  hospital  to  the  processor.  The 
Team  compares  the  discharge  sheets  maintained  by  the  Review  Coordinator 
with  the  discharge  statistics  for  that  hospital  as  compiled  by  the 
processor.     Discrepancies  are  noted  in  the  report  letter  and  could,  if 
sufficiently  serious,  generate  action  from  the  PSRO  Peer  Review  Committee. 

Data  Entry  and  Edit 

Every  abstract  submitted  to  the  PSRO  is  screened  by  the  Data  Clerk. 
Screening  consists  of  checks  for  completeness,  accuracy,  and  reasonableness 
of  concurrent  review  information.     The  Data  Clerk  was  trained  on  an  on- 
going basis  by  the  Data  Manager.     The  edits  performed  by  the  Data  Clerk 
are  relationship  edits  that  specify  logical  relatioships  between  fields 
on  the  abstract.     The  PSRO  Data  Manual  requires  22  relationship  edits 
that  are  performed  by  the  Data  Clerk. 

Data  Processing  and  Output  Analysis 

Prior  to  subcontractor  system  implementation,  the  PSRO  submitted  200 
"live"  abstracts  to  be  subjected  to  processor  edits.     The  error  documents 
generated  by  the  system  were  compared  with  the  original  records  and 
appropriate  changes  made  to  the  edits.     Subsequent  PSRO  submission  of  a 
50,000-record  backlog  uncovered  opportunities  for  further  refinement  of 
the  system. 

In  terms  of  monitoring,  the  PSRO  initially  scanned  all  error  documents 
to  catch  keypunch  errors.     This  system  proved  so  effective  that  the  data 
processor  now  performs  the  same  activity  prior  to  sending  the  error 
documents  to  the  PSRO.     The  PSRO  now  performs  sample  scans  for  keypunch 
errors.     Additionally,  the  PSRO  scans  profile  reports  and  error  documents 
to  detect  programming  problems.       In  general,  weekly  processing  is 
monitored  through  examination  of  reports  and  the  Federal  processor's 
error  reports. 


SUMMARY 

The  South  Carolina  Medical  Care  Foundation  has  a  data  quality  control 
system  that  is  particularly  strong  on  the  front  end:     Review  Coordinators 
receive  formal  initial  and  periodic  training;  the  PSRO  communicates  with 
the  Review  Coordinators  about  particular  problems  or  issues  through 
written  and  filed  procedure  bulletins  and  two  WATS  lines;  the  Validation 
Survey  Team  examines  the  quality  of  data  activities  in  the  hospitals  on 
a  periodic,  operational  basis.     This  system  is  generic  in  the  sense  that 
it  applies  to  both  delegated  and  non-delegated  hospitals. 
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APPENDIX  F 

CASE  STUDY 
WESTERN  MASSACHUSETTS 
PROFESSION  STANDARDS  REVIEW  ORGANIZATION 


PREFACE 


The  following  case  study  descr ibes  the  data  system  and  the  data  quality 
control  program  developed  by  the  Western- Massachusetts  PSRO  to  control 
the  collection  and  processing  of  the  PHDDS  and  optional  data  elements 
in  this  PSRO  area.  This  case  represents  an  example  of  a  PSRO  which 
collects  data  both  through  an  existing  data  collection  system,  and  on 
its  own  data  form  directly  from  some  hospitals. 

We  are  indebted  to  the  Western  Massachusetts  PSRO  for  its  support  and 
cooperation  in  this  quality  control  endeavor,  and  especially  to  Charles 
Everett,  Executive  Director  ;  Sister  Mary  of  Fatima,  Associate  Director; 
Roland  Malboeuf,  Director  of  Information  Systems;  and  Linda  Maloni  and 
Joan  Nodell,  Hospital  Liaisons,  for  their  time  and  assistance,  without 
which  preparation  of  this  case  would  not  have  been  possible. 
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CASE  STUDY 


PSRO;     Western  Massachusetts  PSRO 

PSRO  PERSONNEL  INVOLVED  IN  DATA  ACTIVITIES: 

Charles  Everett,  Executive  Director 

Sister  Mary  of  Fatima,  Associate  Director 

Roland  Malboeuf,  Director  of  Information  Systems 

Linda  Maloni,  Hospital  Liaison 

Bernice  Maclnnis,  Hospital  Liaison 

Joan  Nodell,  Hospital  Liaison 

Paula  McClay,  Data  Coordinator 

Donna  Roosevelt,  Data  Control/Secretary 

Margaret  Porter,  Data  Entry  Operator 

ENVIRONMENT: 

The  Western  Massachusetts  PSRO  covers  approximately  one  third  of  the 
western  part  of  the  State  of  Massachusetts,  and  encompasses  18  acute 
care  hospitals,  all  of  which  are  delegated  for  concurrent  review.  These 
hospitals  discharge  some  55,000  Federal  patients  each  year.    The  PSRO 
collects  data  from  three  hospitals  through  the  Massachusetts  Hospital 
Association  (MHA)   and  its  DIS/CMS  abstracting  service;  the  other  15 
hospitals  complete  the  PSRO's  data  form  and  forward  the  forms  to  the 
PSRO.     The  PSRO's  data  processor  is  the  Dikewood  Corporation,  and  the 
PSRO  uses  Dikewood' s  on- site/ on- line  option,  whereby  data  are  key- 
entered  by  a  PSRO  employee  via  remote  terminal. 

DATA  SYSTEM: 

The  data  system  developed  by  WMPSRO  involves  the  integration  of  both 
tape  input  from  the  data  collector  and  abstract  data  keyed  directly 
into  the  computer  by  the  PSRO.     Data  quality  control  procedures  have 
been  developed  at  several  points  in  the  process,  and  are  planned  at 
further  points.    A  detailed  flowchart  of  the  entire  data  system,  from 
the  time  the  patient  enters  the  hospital  through  the  generation  of 
output  reports,  has  been  developed  and  is  included  on  the  following 
page. 

Activities  Related  to  Data  Abstracting  and  Recording 

The  documentation  and  abstract  completion  processes  are  quite  similar 
among  hospitals  in  the  area.    For  hospitals  which  complete  the  PSRO's 
abstract,  the  information  comes  from  the  Admissions  Office,  and  much 
of  the  abstract  is  completed  concurrently  during  the  patient's  stay. 
Each  Coordinator  maintains  a  worksheet,  which  documents  the  review 
activities  which  were  conducted  and  the  decisions  which  were  made. 
The  Coordinator  completes  the  abstract  and  performs  all  diagnostic 
and  procedural  coding.   The  same  process  applies  in  the  three  hospitals 
which  collect  data  through  MHA,  except  the  Medical  Records  Department 
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EXHIBIT  F-l 


Flow  Diagram:  Western  Massachusetts  PSRO.  Inc. 


EXHIBIT  F-l 


completes  the  abstract  and  does  the  coding,  using  review  information 
from  the  Coordinator's  worksheet. 

Significant  emphasis  is  placed  on  education,  training,  and  monitoring. 
Coordinators  undergo  a  30-40  hour  training  session  initially,  which 
includes  instruction  in  coding  and  abstracting.     These  sessions  are 
primarily  taught  by  the  Hospital  Liaison  personnel,  although  outside 
personnel  have  been  retained  on  occasion  for  coding  and  abstracting 
instruction;  in  the  future,  the  PSRO  intends  to  use  the  expertise  of 
a  new  staff  member  who  is  an  ART,  although  this  person's  primary 
responsibilities  will  be  in  the  area  of  MCE  studies.     In  addition, 
monthly  meetings  are  held  for  all  Coordinators  in  the  area,  at  which 
various  in-service  continuing  education  activities  are  conducted. 
Written  instructions  concerning  the  performance  of  review  activities, 
maintenance  of  Coordinator  worksheets  and  the  information  which  is 
required,  and  the  completion  of  abstracts,  have  been  prepared  and 
distributed  to  all  personnel.    Adherence  to  these  instructions  is 
monitored  in  the  hospitals  on  a  regular  basis  by  the  Hospital  Liaisons. 

The  manner   in  which  diagnoses  and  procedures  are  identified  and 
sequenced  varies  between  hospitals.     Diagnoses  and  procedures  may  be 
identified  from  the  entire  medical  record,  from  the  discharge  summary, 
or  by  the  Coordinator,  based  on  what  he/she  has  been  monitoring. 
Sequencing   is  accomplished   in  a  similar   fashion.     The  PSRO  is  most 
comfortable  with  the  determinations  made  by  Coordinators,  who  have 
been  closer  to  the  case  on  a  regular  basis,  and  who  have  a  better 
understanding  of  what  has  transpired.    Established  procedures  require 
that  questions  about  coding  be  referred  to  the  PSRO  for  resolution. 
Questions  are  logged  to  assure  consistency  in  responses,  and  are  later 
distributed  to  Coordinators  in  the  field. 

Activities  Related  to  Forms  Flow  Control  and  Data  Transfer 

Completed  PSRO  abstracts  are  batched  in  the  hospital  in  quantities  of 
50  and  mailed   in  postage  prepaid  envelopes  to  the  PSRO.     Each  batch 
includes  a  batch  header  sheet  which  lists,  by  document  number,  each 
document  in  the  batch.     The  hospital  retains  a  copy  of  each  batch 
header  sheet.    Batches  are  sequentially  numbered,  and  are  logged  by 
batch  number  and  date  at  both  the  hospital  and  PSRO.     Hospitals  are 
contacted  immediately  to  resolve  any  d  iscrepanc  les  in  numbers  or  totals 
of  abstracts  received. 

The  PSRO  requires  that  abstracts  be  submitted  within  30  days  of  a 
patient's  discharge.    This  is  monitored  through  the  monthly  submission 
of  a  Report  of  Outstanding  Data  Forms.     This  report  identifies  by 
discharge  month  the  number  of  documents  outstanding   for  more  than  30 
days  and  the  reasons  for  the  delay.    Timely  submission  is  met  for  more 
than  90  percent  of  all  Federal  discharges. 

Hospitals  which  submit  abstracts  to  MHA  also  batch  abstracts  in  batches 
of  50.     Abstracts  for  discharges  in  a  given  month  are  due  within  15 
days  of  the  end  of  the  month.    Batches  are  sequentially  numbered  within 
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month,  and  the  last  batch  of  the  month  is  indicated.  Batch 
discrepancies  identified  on  receipt  of  abstracts  are  resolved 
immediately  by  telephone  with  the  hospital. 

Activities  Related  to  Data  Entry  and  Edit 

Batches  of  abstracts  which  have  been  received  are  logged  and  scanned 
for  blank  fields  and  obvious  abstracting  errors  by  the  PSRO.  These 
batches  are  then  forwarded  to  the  key  entry  operator  for  entry  into 
the  computer.     Each  abstract  has  a  pre- printed  document  number,  and 
this  number  is  keyed  twice  to  ensure  accurate  entry.    The  operator 
number  is  written  on  each  abstract  to  verify  that  it  has  been  entered 
into  the  data  base.    If  the  key  entry  operator  detects  ah  error  during 
data  entry,  whether  thr oug h  v isual  examination  or  from  an  edit  message, 
the  error  is  circled  on  the  abstract  and  separated  within  the  batch. 
The  Data  Control  person  logs  the  date  each  batch  has  been  entered  and 
the  number  of  abstracts  with  errors.    All  documents  are  filed  by 
hospital,  by  document  number.    Those  with  errors  are  filed  in  the  error 
file,  and  all  others  are  filed   in  the  permanent  file. 

Bi-weekly  error  listings  are  received  from  Dikewood.   The  PSRO  ver if ies, 
for  each  error  reported,  that  the  error  was  not  a  data  entry  error, 
and  then  sends  one  copy  of  the  appropriate  error  listing  to  each 
hospital.     Error  corrections  are  required   from  the  hospitals  within 
three  days,  and  this  time  frame  is  generally  met.    The  PSRO  data  entry 
operator  keys  corrections  directly  from  the  error  listings.    At  the 
same  time,  the  operator  visually  verifies  that  all  other  information 
in  the  computer  corresponds  to  that  on  the  abstract  which  has  been 
pulled  from  the  error  file.    Once  the  correction  has  been  made,  the 
data  entry  operator  records  her  initials  on  the  abstract  and  forwards 
it  to  the  data  control  person  for  filing   into  the  permanent  file. 

Entry,  edit  and  correction  of  abstracted  data  sent  to  MHA  are  resolved 
between  the  hospital  and  MHA  according  to  previously  established 
procedures.     The  PSRO  requires  that  tape  input  from  MHA  be  sent  to 
Dikewood  within  40  days  after  the  end  of  a  quarter.    MHA  informs  the 
PSRO,  by  letter,  of  the  date  the  tape  was  sent,  the  number  of  documents 
contained  on  the  tape,  the  number  of  documents  withheld  from  inclusion 
on  the  tape,  and   the  reasons  for   such  exclusions.     Errors  in  MHA  data 
detected  by  Dikewood' s  edits  are  resolved   in  exactly  the  same  manner 
as  errors  in  PSRO  input,  through  error  listings  which  are  sent  to  the 
PSRO  corrected  at  the  hospital,  and  re-keyed  by  the  PSRO  key  entry 
operator . 

Activities  Related  to  Data  Processing  and  Output  Analysis 

Data  received  by  Dikewood  are  edited  and  processed  according  to 
contractual  requirements.    Error  listings  are  produced  and  sent  to  the 
PSRO  bi-weekly.    Output  reports,  which  include  reports  from  Dikewood' s 
standard  report  set  and  reports  designed  by  the  PSRO,  are  required  by 
specified  dates  on  a  monthly  basis.     Reports  are  sent  by  certified 
mail,  return  receipt  requested,  so  Dikewood  knows  that  PSRO  has  received 
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everything  sent.     The  PSRO  compares  the  error  listings  to  the 
exact  count  of  abstracts  entered  by  the  PSRO  and  forwarded  by 
MHA,   so  it  knows  that  everything  transmitted  has  been  received. 
The  End  of  Quarter  Error  Listing  lists  all  documents  in  the 
computer  with  discharge  dates  in  the  given  quarter,  and  these 
totals  are  checked  against  the  totals  on  the  output  reports. 

DATA  QUALITY  CONTROL  SYSTEM; 

The  data  quality  control  system  developed  by  WMPSRO  has  both  an 
on-going  concurrent  quality  monitoring  component,  and  a  retro- 
spective monitoring  component.     Several  parts  of  each  component 
are  already  in  place,  and  others  are  under  development  or  planned 
for  future  implementation. 

Quality  Control  Activities  Related  to  Data  Abstracting  and 
Recording 

Quality  control  related  to  data  collection  in  the  hospital  begins 
in  the  PSRO's  training  programs,  and  is  an  integral  component  of 
the  Hospital  Liaison  function.     The  integrity  of  coding  and  ab- 
stracting activities  is  verified  during  routine  visits  to  each 
hospital.     Screening  of  abstracts  for  completeness  and  accuracy 
is  usually  performed  by  two  persons  (the  Coordinator  and  the  clerk, 
or  the  abstractor  in  the  Medical  Records  Department)  before  ab- 
stracts leave  the  hospital.     Following  receipt  by  the  PSRO,  absracts 
are  also  screened  manually,  primarily  for  completeness  and  for 
validity  of  review  data.     Other  editing,  including  that  for  accuracy 
of  diagnoses  and  procedures,  is  left  to  the  computer. 

Retrospective  analysis  of  the  quality  of  coding  and  abstracting 
is  a  Hospital  Liaison  staff  function.     Once  each  quarter,  Dike- 
wood  selects  a  sample  of  40  records  from  each  hospital  submitting 
data  tapes  and  generates  a  facsimile  of  the  PSRO's  abstract  for 
each  discharge.     From  these,  the  PSRO  selects  a  sub-sample  of  10  - 
40  records  per  hospital  for  on-site,  indepth  review.     For  hospitals 
which  submit  data  forms,  the  PSRO  selects  a  sub-sample  (10  -  40 
records)  of  the  sample  identified  by  the  fiscal  intermediary  for 
post-payment  monitoring.     The  PSRO  selects  these  records  for 
in-depth  review  to  minimize  the  effort  that  the  hospital  must 
expend  in  locating  records,  and  to  compare  monitoring  results 
by  the  fiscal  intermediary.     The  hospital  is  notified  prior 
to  the  visit,  and  pulls  the  appropriate  documents  for  review  by 
the  Liaison  personnel. 

During  on-site  visits  to  hospitals  which  complete  the  PSRO's 
abstract,  the  Liaison  personnel  compare  the  abstract,  Coordinator's 
worksheet,  and  Medical  Record  to  each  other,  and  look  for  any  dis- 
crepancies which  may  exist.     In  those  hospitals  which  complete  MHA 
abstracts,  the  printout  generated  by  Dikewood,  Coordinator's  work- 
sheet, and  medical  record  are  compared.     For  each  discharge,  Liaison 
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personnel  complete  a  form  containing  questions  about  the  con- 
sistency and  correctness  of  the  coding  and  abstracting  process. 
Instances  of  isolated  error,  as  well  as  patterns  of  recurring 
errors,  are  examined  and  documented. 

Site  visits  by  Liaison  personnel  for  formal  purposes  are  con- 
ducted at  least  once  each  month  in  each  facility.  Following 
each  visit  a  report  (generally  written,  although  sometimes 
verbal  if  no    areas  of  difficulty  are  found)   is  issued  to  the 
hospital,  with  copies  to  Administration  and  the  U.R.  chairman. 
General  problem  areas  identified  are  raised  at  the  next  monthly 
Coordinator's  meeting,  for  the  benefit  of  all  Coordinators  in 
the  PSRO  area.     Follow-up  education  programs  (for  Coordinators, 
physicians,  or  whomever)  are  arranged  as  indicated  by  the  problem 
identified.     In  addition,  Coordinators  are  re-tested  once  each 
year  to  assure  that  appropriate  levels  of  review,  coding,  and 
abstracting  skills  are  maintained.     Efforts  to  assure  still  more 
uniformity  in  the  coding  and  abstracting  processes  are  under 
consideration. 

Quality  Control  Activities  Related  to  Forms  Flow  Control  and 
Data  Transfer 

The  PSRO  believes  that  it  receives  abstracts  for  all  Federal 
admissions.     Logs  of  all  Federal  admissions  are  maintained  by 
each  Coordinator,  and  collected  monthly  by  hospital  liaison 
personnel.     Information  in  these  logs  is  compared  to  the  number 
of  abstracts  submitted  by  each  hospital  to  the  PSRO  Data  Coordina- 
tor.    In  addition,  all  hospitals  are  delegated  and  will  be  reimbursed 
for  review  costs,  on  a  per-admission  basis,  by  the  fiscal  inter- 
mediary.    The  PSRO  plans  to  request  from  the  intermediary  monthly 
figures,  by  hospital,  on  the  number  of  admissions  reported,  which 
will  serve  as  a  further  cross-check  to  verify  that  abstracts  are 
received  for  all  patients.  The  PSRO  also  believes  that  no  bills 
are  certified  for  payment  without  a  corresponding  abstract  being 
prepared,  and  feels  that  the  intermediary  monitors  the  presence  of 
certifications  closely  for  payment  purposes.     Although  hospital 
rules  preclude  filing  a  chart  until  it  is  abstracted,  this  is 
not  yet  stipulated  in  MOUs,  and  the  PSRO  cannot  be  positive  that 
abstracts  are  submitted  to  the  data  collector  for  all  Federal  dis- 
charges.    A  monitoring  method  involving  comparison  of  discharges 
reported,  by  month,  by  the  data  collector,  against  admissions 
billed  to  the  fiscal  intermediary,   is  under  development. 

The  PSRO  generally  understands  the  information  flow  within  each 
of  its  hospitals,  and  plans  to  document,  in  detail,  the  manner 
in  which  PSRO-related  data  move  in  each  facility.     This  will 
help  assure  consistency  in  quality  control  procedure  development, 
and  will  greatly  facilitate  identification  of  problem  areas  when 
data-related  difficulties  appear  in  any  hospital. 
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The  control  procedures  established  for  data  transfer  are  complete 
where  PSRO  abstracts  are  concerned,  and  there  are  no  apparent 
possibilities  of  losing  documents.     Both  the  PSRO  and  hospital 
have  copies  of  the  batch  header  sheets,  which  identify  individual 
documents.     Batches  are  sequentially  numbered,   so  that  if  any 
batch  is  missing,  this  will  become  apparent  as  soon  as  the  subse- 
quent batch  is  received;   the  PSRO  also  anticipates  an  approximate 
volume  of  documents  from  each  hospital  in  during  each  month. 
Abstracts  have  pre-printed  document  numbers,  and  the  PSRO  records 
the  documents  delivered  to  each  hospital.     Copies  of  abstracts 
are  filed  in  the  hospitals  by  document  number  and  cross-referenced 
by  patient  name  to  facilitate  location  of  any  documents.  There 
have  been  no  problems  with  lost  documents. 

Where  tape  input  is  to  be  transferred  from  the  data  collector 
to  the  data  processor,  the  PSRO  obtains  an  error  listing  from 
Dikewood  two  weeks  following  receipt  of  such  input,  and  currently 
uses  this  method  to  monitor  processor  receipt  of  tape  data.  The 
PSRO  currently  monitors  anticipated  discharge  volume  from  each 
such  hospital  against  the  number  of  discharges  reported  by  MHA 
in  the  tape  transmittal  letter  which  is  sent  to  the  PSRO. 

Quality  Control  Activities  Related  to  Data  Entry  and  Edit 

Control  procedures  concerning  receipt,  verification,  and  logging 
of  data  have  been  described  above.     All  data  on  abstracts  are 
keyed  into  the  computer,  regardless  of  error  messages  during 
entry  (errors  are  circled  on  abstracts,  and  these  documents  are 
filed  in  a  separate  error  file  for  subsequent  correction) , 
except  in  instances  where  the  error  prohibits  entry  to  the  file 
(such  as  an  invalid  hospital  identifier).     These  errors  are 
corrected  with  the  assistance  of  the  Data  Coordinator,  Hospital 
Liaisons,  and  hospital,  if  necessary,  and  data  entry  continues. 

Once  each  month  the  Director  of  Information  Systems  pulls  approxi- 
mately 15  records  from  the  file  on  a  sample  basis,  recalls  those 
documents  onto  the  terminal,  and  compares  the  data  in  the  data- 
base with  the  hard  copy  document  for  completeness  and  accuracy. 

Edit  listings  are  received  from  Dikewood  every  two  weeks  for  all 
data,  including  input  from  the  data  collector.     These  reports  are 
reviewed  to  identify  any  keying  errors,  and  then  a  copy  of  the 
appropriate  pages  is  sent  to  each  hospital.     Corrections  are 
made  on  the  report,  and  are  returned  to  the  PSRO  within  three 
days.     Each  error  record  is  called  onto  the  CRT  terminal,  and 
the  appropriate  corrections  are  made.     For  errors  made  on  PSRO 
abstracts,  the  abstract  is  pulled  from  the  error  file,  and 
visual  verification  is  made  that  the  entire  record  is  now 
accurate.     The  abstract  is  then  filed  by  hospital,  by  document 
number,  in  the  permanent  file. 
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Quality  Control  Activities  Related  to  Data  Processing  and  Output 
Analysis 

The  PSRO  monitors  closely  the  timeliness  and  quality  of  processor 
outputs,  including  error  listings,  routine  reports,  and  tape 
submissions  to  HSQB,  on  a  regular  basis.     The  PSRO  tested  the 
processor's  data  system  with  a  test  data  set  consisting  of  40 
abstracts.     The  data  collector  also  prepared  a  test  data  tape 
to  facilitate  testing  of  procedures  to  receive  tape  input,  and 
to  assure  that  there  were  no  hardware  compatibility  problems. 
A  further  test  was  conducted  with  live  data  prior  to  the  actual 
commencement  of  processing  activities. 

The  PSRO  has  (physically)  reviewed  and  approved  the  internal 
data  flow  control  system  of  the  data  processor,  but  it  does 
not  maintain  on-going  quality  review  procedures  concerning 
the  internal  operations  of  either  the  data  collector  or  the 
data  processor.     The  PSRO  is  currently  attempting  to  develop 
procedures  to  verify  that  all  abstracts  transmitted  and  received 
by  Dikewood  are  actually  reflected  on  the  output  reports.  This 
is  made  difficult  because  bi-weekly  error  listings  (which  are 
used  to  verify  that  all  data  have  been  received  by  Dikewood) 
are  generated  by  date  of  entry,  and  output  reports  are  generated 
by  date  of  patient  discharge.     However,  this  difficulty  does 
not  exist  with  tape  submitted  by  the  data  collector  since  infor- 
mation on  the  tape  is  sequenced  by  date  of  discharge.     It  may  be 
possible  to  effect  software  modifications  to  enable  counting 
documents  entered  by  month  of  discharge. 

SUMMARY 

The  data  quality  control  system  developed  by  the  Western 
Massachusetts  PSRO  is  generally  thorough  and  comprehensive. 
Coding  and  abstracting  procedures  are  generally  uniform  through- 
out the  area,  and  the  PSRO  places  great  emphasis  on  the  importance 
of  these  activities.     The  PSRO  is  reasonably  certain  that  it 
receives  data  for  each  Federal  patient  discharged  from  its 
hospitals . 
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APPENDIX  G 


CASE  STUDY 
CALIFORNIA  AREA  XX  PSRO 


I 


CALIFORNIA  AREA  XX  PSRO 


This  case  study  describes  the  data  quality  program  procedures 
being   implemented  by  the  Area  XX  PSRO,  Sherman  Oaks,  California.  The 
development  of  this  program  coincided  with  the  commencement  of  review 
in  early  1977. 

We  wish  to  express  our  gratitude  for  the  cooperation  received 
during  the  preparation  of  this  document.     Special  thanks  are  due  the 
Executive  Director,  Lila  Marcus,  the  Data  Manager,  Paul  Schmitz,  and 
the  Area  XX  Board  of  Directors  and  Data  Committee. 

INTRODUCTION 

The  California  Area  XX  PSRO  covers  the  San  Fernando  and  Antelope 
Valleys  of  Los  Angeles  County.     Within  this  area  are  36  acute  care 
facilities,  ranging   in  size  from  13  to  about  500  beds,  and  65  long- 
term  care  facilities.     There  are  approximately  80,000  annual  Federal 
patient  discharges  from  the  acute  care  facilities.     Currently, -only 
Medicare  patients  are  being  reviewed  as  a  Memorandum  of  Understanding 
has  not  been  executed  with  the  State  Medicaid  Agency. 

At  this  time,  delegated  review,  which  began  in  March  1977,  has 
been  implemented  in  32  of  the  36  hospitals.    It  is  hoped  that  delegated 
review  would  be  implemented  in  the  remaining  four  hospitals,  but  there 
are   indications  that  one  or  two  may  be  assigned  non-delegated  status. 

Thirty-one  of  the  hospitals  subscribe  to  one  of  the  five  discharge 
abstract  systems  operating   in  the  area;  one  hospital  has  an  internal 
system  and  four  will  complete  Area  XX  PSRO  hard  copy  abstracts.  A 
copy  of  this  abstract  is  shown  as  Exhibit  I. 

Each  hospital  uses   its  own  review  coordinator  worksheet  -  the 
source  for  the  "P"  elements  -  but  hospitals  are  required  to  collect 
uniform  information  for  PSRO  monitoring  purposes.     In  addition,  a 
Summary  Review  Log    (as  shown  in  Exhibit  II)    is  completed  by  each 
hospital  and  forwarded  to  the  PSRO  each  week.     This  Log   is  used  for 
various  purposes  including; 

Review  monitoring 

Review  activity  analyses 

HSQB/121  preparation 

Federal  patient  counts 

Assuring  that  all  cases  certified  for  billing  have,  in  fact, 
been  reviewed. 
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CONTROL  NO. 


CALIFORNIA  AREA  XX  PSRO 
DATA  ABSTRACT 


EXHIBIT  G-l 


ADMISSION/ D I S CHARGE   I NFORHAT I ON 


2.  Hospital  Number 

LlLlu 


3.  ffedical  Record  Number 
LiJj-XJ 


4.  Medicare  Number 


'111 


5.  Birthdate 
tonth 


Day 


aJ  U 

Year    Over  100 
Years-*--' 


8.  Residence 

1     >   ,1    1  t 

Zip  Coda 

11.  Class  of  Admission 

□  1-Emergency 

□  2-Elective 

13.  Discharge  Date 

U  i'l  '  i 

.Month     Day  Year 
15.  Attending  Physician 

1  i  iri 

17.  Diagnosis 
a.  1    J  I 


6.  Sex:  □  1-Male 

□  2-Female 

9.  Source  of  Payment 

□  03-Medicare 

□  04-Medi-Cal 

□  05-Title  V 

D      ^-Other :  Specify 


J!  i  i  i  i  i  i  i  i  i  i.f 


Principal 


c. 
d. 
e. 


1     1  1 


I     I  I 


14.  Disposition  of  Patient 

□  01-Other  Hospitals 

□  02-SNF 

□  03-ICF 

□  04-Other  Institution 

□  05-Home 

□  06-Home  Health 

□  07-LAMA 

□  08-Died,  Autopsy 

□  09-Died,  No  Autopsy 


7.  Race:  □  0-Unknown 

□  1-White 

□  2-Black 

□  3-Other 

10.  Admission  Date  Hour 

I  i  I  ■  I  i  I  l_i_J 

Month     Day  Year 

12.  Source  of  Admission 

□  1-Routine 

C  2-Emergency  Room 

□  3-Other  Hospital  (Acute) 

□  4-SNF-ICF 

□  5-Other  Facility 

□  6 -Other 

16.  Operating  Physician 

l—i—l— J 

18. 
a. 

b. 


lilt 


Procedure  and  Dates 

jJl  i 

Principal 


Month  Day  Ye 


c.  l 
L 


ADMISSION 


■ICATTON  INFORMATION 


19  |    |    1  Number  of  Days  Assigned  at  Admission 


20.  Admission  Certification 
Process: 


□  2 -Concurrent  admission  review certification  granted 
Q  3 -Concurrent  admission  review certification  denied 

□  4 -Concurrent  admission  review:was  not  performed  or  retrcspe 


21.  Basis  for  Assignment  of  Initial 
Length  of  Stay:  □  0-Not  required 

□  1-PSRO  norms  used 


22.  Admission  Certification  Level  of 
Review:  □  0-Not  required 

Q  1-Review  Coordinator 
□  2-  Physician  Advisor 


CONTINUED  STAY  REVIEW  (CSR)  INFORMATION 


23 •  1    11    ,1  Total  Days      24.  |  J  Total  Number  of    25.  ]  [Total    26.  J  ]  Total 

Certified  (Assigned)  Reviews  Referred  to  Extensions         Extensions  Denied 

Physician  Approved 


27.  Unspecified:  Study  No:|    |   |a.[    1    [b.|  |    jc.|     ]    |d.  1     1    \e.\    |    |f.|    I    |g.|     I     |h.  | 
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SYSTEM  DESCRIPTION 


The  basic  configuration  of  the  Area  XX  data  system  is  for 
hospitals  subscribing  to  an  abstract  system  and  the  hospital  with  an 
internal  system  to  forward  tapes  directly  to  Dikewood  Industries,  the 
processor  selected  by  Area  XX  in  August  1977.     Hospitals  using  the 
PSRO  abstract  submit  these  to  the  PSRO  and,  in  turn,  they  are  forwarded 
to  the  processor  for  entry  into  the  data  base. 

In  the  remainder  of  this  section,  an  overview  of  the  data  system 
is  provided.     A  schematic  of  the  system  flow  is  shown  in  Exhibit  III. 

Activities  Related  to  Data  Abstracting  and  Recording 

In  almost  all  instances,  medical  records  personnel  are  responsible 
for  abstract  preparation.    A  supplemental  form  to  record  the  "P" 
elements   is  completed  for  all  Federal  patients  by  the  review 
coordinator  and  forwarded  to  medical  records  where  the  data  is 
transcribed  onto  the  particular  system  abstract  being  used.    This  form, 
an  example  of  which  is  shown  in  Exhibit  IV,  is  tailored  to  each  system 
item  related  to  the  "P"  elements,  and  to  the  particular  coding  structure 
used  by  each  system.     Hence,  there  are  seven  versions  of  this  form. 

In  addition,  a  supplemental  training  manual  has  been  prepared  for 
each  system  which  describes  the  PSRO  data  set,  its  relationship  to 
each  system  abstract,  and  the  methods  to  be  employed  for  reporting 
each  data  element  via  the  system  in  use  by  the  hospital. 

Training  in  abstract  preparation  is  provided  hospital  staff  on 
an  individual  basis  at  the  time  review  is  implemented.     This  is 
discussed  in  more  detail  in  later  sections  related  to  data  quality 
control  procedures. 

Three  approaches  are  employed   in  routing  the  data,  once  the 
abstracts  are  coded: 

The  hospital  with  an  internal  system  forwards  a  tape  directly 
to  the  processor 

Abstracts  are  submitted  to  data  collection  systems  (also 
referred  to  as  abstract  systems  or  agencies)  which,  in  turn, 
forward  tapes  to  the  processor  when  corrections  are  received 

Hospitals  completing  the  Area  XX  abstracts  forward  these  to 
the  PSRO  where  they  are  checked,  batched,  recorded,  and  sent 
to  the  processor  for  processing. 

Activities  Related  to  Forms  Flow  Control  and  Data  Transfer 

There  are  four  aspects  of  forms  control  and  data  flow  monitoring. 
Each  is  briefly  discussed  below  and   is  elaborated  upon  in  a  later 
section  where  quality  control  procedures  related  to  data  flow 
monitoring  are  discussed. 
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Hospital  Data  Submission 


Each  time  a  hospital  forwards  the  final  batch  of  abstracts 
for  a  given  discharge  month  to  the  abstract  system,  it  is 
required  to  notify  the  PSRO.    Similarly,  hospitals  completing 
Area  XX  hard  copy  abstracts  indicate  "final  batches"  when 
submitting  these  to  the  PSRO. 

Abstract  System  Data  Submission 

Each  month,  collection  systems  are  required  to  report,  by 
hospital  and  discharge  month,  all  data  submitted  to  Dikewood 
during  the  month. 

Processor  Submissions 

When  tapes  are  forwarded  from  Dikewood  to  HSQB,  a  report  on 
hospitals,  months  inlcuded,  and  volume  of  discharge  is  filed 
with  the  PSRO. 

Summary  Control  Log 

Based  on  the  three  types  of  reports  described  above,  a 
Summary  Control  Log   is  prepared  by  the  PSRO  to  track  the 
data  flow  through  the  system.     The  Log  is  also  used  to 
identify  delinquent  reporting  and  verify  discharge  volume 
counts  reported  from  various  points  in  the  system. 

Activities  Related  to  Data  Entry  and  Edit 

Because  existing  abstract  systems  are  the  basis  for  the  Area  XX 
data  system,  the  PSRO  is  not  directly  involved  in  the  data  entry 
process.     Due  to  the  fact  that  only  three  optional  items  are  being 
collected,  the  basic  edits  contained  in  Transmittal  28  and  50  are  the 
PSRO  standards.    However,  some  collection  systems  to  which  the  Area  XX 
hospitals  submit  their  data  do  use  more  stringent  edits. 

In  instances  where  hospitals  subscribe  to  an  existing  abstract 
system,  correction  of  errors  detected  by  the  edit  routines  are  handled 
directly  between  the  system  and  the  hospital.  For  the  four  hospitals 
using  the  Area  XX  abstract,  error  listings  from  Dikewood  are  forwarded 
to  the  PSRO,  the  corrections  are  obtained  from  the  hospital,  and  these 
are  resubmitted  to  the  processor  via  the  PSRO. 

Activities  Related  to  Data  Processing  and  Output  Analysis 

Processing  is  undertaken  using  Dikewood's  standard  processing 
procedures.    The  one  aspect  of  the  Area  XX  system  which  is  not  routine 
for  Dikewood  is  the  multiple  tape  merging  which  is  required  to  combine 
data  from  the  various  collectors  and  from  the  hospitals. 
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Error  reports  are  generated  on  a  hospital  by  hospital  basis  and 
forwarded  to  the  PSRO  after  the  processor  edits  the  data.  The  initial 
reports  were  used  primarily  to  identify  editing  and  formatting  problems 
at  the  abstract  system  level.  Except  in  the  case  of  one  collection 
system,  readily  identifiable  patterns  were  evident  and  the  corrections 
have  been  effected.     The  types  of  problems  encountered  included: 

One  abstract  agency  forwarded  all  Federal  patient  discharges 
for  a  month;  however,  since  review  by  Area  XX  is  only 
performed  on  admiss ions,  some  of  the  discharges,  especially 
during  the  first  month  of  review  in  a  hospital,  related  to 
admissions  of  earlier  months  which  were  not  reviewed. 

One  abstract  system  only  collected  day  and  month  of 
procedures  and  defaults  the  year  by  comparison  of  admission 
and  discharge  dates;  this  default  did  not  work  properly  for 
one  procedure  field. 

No  physician  identifier  was  submitted  on  principal  "non- 
surgical" procedure  field. 

Confirmation  that  the  corrections  to  these  problems  have  been «made 
will  be  shown  by  the  results  of  processing  first  quarter  1978  data. 

To  date,  only  one  set  of  routine  reports  has  been  received  by  the 
PSRO  from  Dikewood  because  the  system  has  been  operable  only  since 
late  1977.     I     is  anticipated  that  these  reports  will  be  analyzed  by 
the  PSRO's  data  staff,  then  reviewed  by  the  PSRO  Data  Committee. 
Following  this  review  the  reports  will  be  provided  to  other  PSRO 
committees  to  support  their  activities. 

DATA  QUALITY  CONTROL  SYSTEM 

The  procedures  to  be  discussed  in  this  and  following  sections  of 
the  case  study  are  the  core  of  the  Area  XX  data  quality  luuliu!  program. 
The  objectives  of  this  program  can  be  summarized  as  follows: 

To  minimize  the  potential  for  data  quality  problems  to 
develop 

To  determine  if  data  quality  problems  exist  at  any  point  in 
the  system  and,  if  so,  where  they  exist 

To  identify  the  nature  of  the  problem 

To  determine  if  these  problems  are  associated  with  the  data 
collection  process,  specific  abstract  agencies  or  personnel 
within  these  agencies 

To  develop  approaches  for  remedial  or  follow-up  action  and 
for  provision  of  technical  assistance  to  hospitals  as 
necessary. 
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Approaches  to  satisfying  these  objectives  are  discussed  in  the 
sections  which  follow.  The  mechanisms  to  be  described  are  used  by  the 
data  staff  to  identify  problem  areas  and  to  seek  their  resolution  from 
the  agencies  involved.  In  those  instances  where  resolution  is  not 
forthcoming  in  a  timely  manner,  these  procedures  are  used  as  the  basis 
for  reporting  to  the  Data  Committee,  the  general  activities  of  which 
are  described  in  the  final  section  of  this  paper.  It  is  this  Committee 
which  is  responsible  for  instituting  formal  action  when  necessary. 

Pre-Delegat ion  Activities 

Activities  related  to  data  quality  control  begin  at  the  time 
delegation  of  a  hospital  is  being  considered.    Prior  to  delegation,  a 
survey  is  made  of  each  hospital.     The  specific  data-related  issues 
addressed  in  the  survey  include: 

The  discharge  abstract  system  being  used 

Current  abstract  backlog 

The  coding  system  being  used. 

The  survey  is  followed  by  an  on-site  visit  by  members  of  the  PSRO 
Delegation  Committee  at  which  time  any  problems  identified  during  the 
survey  are  discussed.  For  instance,  if  there  is  a  significant  abstract 
backlog  which  cannot  be  eliminated  prior  to  the  proposed  implementation 
of  PSRO  review  in  the  hospital,  the  PSRO  may  request  that  the  hospital 
complete  a  PSRO  abstract.  This  would  assure  timely  and  accurate 
completion  of  PSRO  data  until  such  time  as  the  backlog  has  been 
eliminated  and  the  hospital  has  demonstrated  the  capability  to  collect 
PSRO  data  in  a  timely  manner  on  the  hospital's  abstract. 

Pre-Implementa t ion  Activities 

Before  implementation,  several  data  related  activities  are 
discussed  with  the  appropriate  hospital  staff  and  include: 

In-service  training  related  to  the  Review  Activity  Summary 
Log 

Preliminary  training  in  abstracting  procedures  using  the 
hospital  data  system  manual  supplemented  by  a  PSRO  training 
manual  designed  to  relate  to  each  abstract  system,  and  a 
discussion  of  the  trailer  abstract  used  to  initially  record 
the  "P"  elements   (Exhibit  IV). 

A  discussion  of  the  flow  of  all  data  in  the  PSRO  including 

Concurrent  review 
-        Data  collection 
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Data  abstracting 

Data  transmission 

MCE  study  reporting 

Determination  of  how  the  review  coordinator  worksheet  data 
corresponds  with  medical  record  admission  and  discharge  data 

Discussions  on  confidentiality 

Requests  for  a  data  release  agreement  when  the  hospital 
proposes  to  use  a  discharge  abstract  system. 

Post-Implementation  Activities 

Immediately  following  implementation,  an  intensive  training 
session  is  held  with  the  review  coordinator  and  medical  records 
personnel.     It  is  felt  that  holding  this  session  after  two  or  three 
days  of  review  experience  provides  a  realistic  framework  to  which 
hospital  personnel  can  relate.     Topics  covered  include: 

Intensive  review  of  the  abstract  and  data  training  manual 

Review  of  the  trailer  information 

Review  of  hypothetical  coding  situations. 

At  this  time,  a  listing  of  all  physicians  and  their  assigned  hospital 
identification  numbers  is  obtained. 

During  a  subsequent  visit,  a  review  is  undertaken  of  the  first 
50  abstracts  prepared.  Findings  are  recorded  on  a  sheet  which  lists, 
by  case,  the  individual  data  elements  and  coding  problems  identified. 
The  objectives  of  this  screening  are  to  verify  that  the  appropriate 
data  fields  are  filled,  to  review  the  accuracy  of  the  abstracting,  and 
to  verify  that  concurrent  review  data  is  displayed  for  those  patients 
reviewed. 

The  general  findings  are  discussed  with  the  staff  and  documented. 
Where  necessary,  additional  training  is  performed. 

Quality  Control  for  Data  Flow  Monitoring 

Five  mechanisms  are  used  for  the  data  flow  monitoring  functions  in 
the  Area  XX  quality  control  program: 

-        Hospital  data  submission 

Abstract  system  data  submission 
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-        Processor  submissions 

Summary  Control  Log 

Medical  Record  Control  Index. 
Each  of  these   is  described  below. 

Hospital  Data  Submission  Monitoring 

Each  time  a  hospital  submits  the  final  batch  for  a  discharge 
month,  a  form  letter  as  shown  in  Exhibit  V  is  sent  to  the 
PSRO.     These  data  are  entered  on  the  Summary  Control  Log 
(discussed  below).     Final  batches  are  required  to  be 
submitted  within  30  days  from  the  end  of  the  discharge  month. 

Abstract  System  Data  Submission  Monitoring 

Within  45  days  of  the  end  of  the  quarter,  each  abstract 
system  is  required  to  submit  a  report  on  data  forwarded  to 
Dikewood  and  list  the  discharge  months  by  hospital.  This 
report  is  shown  in  Exhibit  VI.     A  refinement  being  , 
implemented  is  to  list  the  actual  number  of  discharges 
against  each  entry,  in  order  to  track  the  number  of  abstracts 
for  each  month  which  may  have  been  held  back  due  to  the 
errors  not  having  been  resubmitted.     These  figures  will  be 
matched  against  hospital  review  volumes  on  the  Summary 
Review  Logs  (Exhibit  II).     Currently,  an  estimate  of 
corrected  batches  can  be  obtained  only  from  those  systems 
which  withhold  a  complete  month  until  all  corrections  are 
made.     It  is  felt  that  the  additional  volume  reporting  will 
enhance  the  overall  data  flow  monitoring  system  by 
identifying  partial  submissions  being  forwarded  to  the 
processor. 

Processor  Submissions 

Due  to  the  early  system  problems  discussed  above,  only  one 
tape  has  been  forwarded  to  HSQB.  However,  it  is  felt  that 
the  problems  have  been  corrected  and,  on  future  submissions, 
the  processor  will  report  to  the  PSRO  the  date  of  submission 
of  the  PHDD  tapes  and  the  volume,  by  hospital,  for  the  months 
involved. 

Summary  Contrpl  Log 

A  Summary  Control  Log  is  used  as  the  primary  control  document 
for  all  data  flow.     It  is  prepared  by  the  PSRO  from  the 
inputs  of  hospitals,  abstract  systems,  and  the  processor,  as 
described  in  the  preceding  sections.     A  facsimile  of  this 
Log  as  it  will  appear  with  the  additional  volume  reporting 
requirements  is  shown  in  Exhibit  VII. 
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TRAILER    ABSTRACT  EXHIBIT  G -4 

CONCURRENT  REVIEW  SUMMARY  INFORMATION 


Patient  Name 


Discharge  Date_ 


Medical  Record  Number 


ABSTRACT 
FIELD  NUMBERS 


FIELD 
CODING 

/  / 


I    I    I  I 


I  I 


I  I 


I  I 


I    I    I    I  I 


I    I  I 


PSRO  DATA  ELEMENT  DESCRIPTIONS 

Admission  Certification  Process 

2=Concurrent  admission  review:  certification  granted 
3=Concurrent  admission  review:  certification  denied 
4=Concurrent  admission  review:  retrospective 

Days  Assigned  at  Admission 
000  to  999 

Basis  for  Assignment  of  Initial  Length  of  Stay 

0=Not  Required 
1=PSR0  norms  used 

Admission  Certification  Level  Review 

0=Not  Required 
l=Review  Coordinator 
2=Physician  Advisor 

Total  Number  of  Reviews  Referred  to  Physician  Advisor 
0  to  9 

Total  Number  of  Days  Certified  Assigned 
0000  to  0999 

Extension  Denial 


00=None 

01=0ne  or  more 


Note: 


In  recording  PSRO  Data  Element  Codes  it  is  important  to  begin  at  the 
right  hand  side.  All  blank  spaces  must  be  filled  with  zeroes,  e.g., 
9  Day  Assigned  Admission  would  be  recorded  as  /0/0/9/. 


EXHIBIT  G-5 

[joikitfa  u^ftea  XX 

^P/to^ssionaC  Standards  Qei/ieu;  ©Agpiugatton 

SAN  FERNANDO  VALLEY  AND  ANTELOPE  VALLEY  PSRO 


TO: 


DATA  MANAGER 
CALIF.  AREA  XX  PSRO 
15250  VENTURA  BLVD. 
SUITE  810 

SHERMAN  OAKS,   CALIF.  91403 


FROM: 


SUBJECT: 


HOSPITAL  NAME 


SUBMISSION  OF  ABSTRACTS 


Our  final  batch  of  abstracts  for  the  month  of  _ 
has  been  mailed  to  our  computer  service  on  this 
date 


Medical  Records  Persons  Name 


EXHIBIT  G-6 


(Your  letter  Stationery) 


To:  Dikewood  Industries 

Attention:     M.  Shafi 
1009  Bradbury  Drive,  S.E. 
Albuquerque,  New  Mexico  87106 

From:       (Name  of  sender  or  person  to  contact) 

Re:  Data  Tape  for  Area  20  PSRO  9  CA  20 

Hospital  Patient  Discharges 

The  enclosed  data  tape  contains  patient  discharges  from  hospitals 
in  Area  20  PSRO. 

Date  Tape  was  Sent  

Number  of  Records 


Number  of  Reels 


I»ist  of  Hospital  ID  Number  and  Months  Reporting  on  Tape 

Hospital  ID  No.  Months  Hospital  ID  No.  Months 

or  Name  Reporting  or  Name  Reporting 


Please  return  previous  submitted  magnetic  tape. 

cc:     Data  Manager 

California  Area  20  PSRO 
15250  Ventura'  Boulevard 
Sherman  Oaks,  California  91403 


REV. 3-78 
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By  comparing  the  Summary  Control  Log  and  the  Summary  Review 
Log   (completed  by  the  hospitals  and  shown  as  Exhibit  II),  and 
PSRO  can  identify  major  data  flow  problems  including: 

-  Hospital  abstract  submission  delays 
Error  correction  cycle  delays 

-  Abstract  system  processing  delays 
Processor  delays 

Patient  count  discrepancies 

-  Reconciliation  of  patient  volumes  at  each  point  in  the 
system  operations. 

A  check  on  Federal  patient  counts  is  maintained  by  the  use 
of  the  Summary  Review  Log  prepared  by  the  hospital  (Exhibit 
II).  These  figures  are  checked  against  abstract  counts  and 
discharges  reported  by  the  various  collection  agencies.  The 
log  is  also  used  to  compile  the  HSQB/121,  and  to  develop 
other  review  statistics  within  the  PSRO.  A  control  chart 
which  lists  the  submissions  of  each  hospital  by  individual 
weeks  is  maintained  by  the  PSRO  to  track  receipt  of  the  Logs. 

Medical  Record  Control  Index 

Currently,  a  Medical  Record  Control  Index  is  generated  by 
the  processor  when  tapes  are  received  from  collection 
agencies.     This  lists  by  hospital  and  patient: 

Date  of  discharge 

System  control  number 

Medical  record  number. 

This  index  has  been  used  to  verify  patient  counts  and,  where 
necessary,  to  track  records  which  appear  to  be  missing.  With 
implementation  of  the  new  volume  reporting  procedures 
discussed  above,  use  of  this  log  may  be  discontinued. 

Data  Input  Quality  Control 

Under  the  terms  of  the  subcontract,  the  processor  was  required  to 
conduct  a  system  test  of  the  data  processing  system.     This  test  was 
designed  and  performed  by  Dikewood  in  October  1977. 

Tests  of  the  collection  systems  were  not  performed  in  the  interest  of 
time  and  since  these  systems  were  serving  other  PSROs,  few  problems 
were  anticipated.     In  retrospect,  however,  such  tests  should  have  been 
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performed.     As  discussed  above,  several  problems  in  coding  and 
formatting  were  encountered  which  caused  about  a  four  month  delay  in 
operational i zi ng  the  system;   a  test  of  the  collector  systems  might 
have  alleviated  some  of  these  problems. 

As  will  be  discussed  below  in  the  section  related  to  data  validation, 
checks  are  to  be  made,   in  the  future,  on  key  entry  accuracy  and 
reformatting  integrity. 

Report  Monitoring  Control 

Several  report  monitoring  logs  are  maintained  to  generate: 

Monthly  error  reports 

Monthly  hospital  reports 

Quarterly,  semi-annual,  and  annual  reports. 

These  are  used  to  maintain  the  reports  flows  against  subcontract 
deliverable  schedules,  as  described  below: 

Monthly  Error  Reports 

The  monthly  error  reports  are  designed  to  track  errors 
identified  by  the  processor  in  editing  data.     The  error 
listings  were  particularly  important  when  initial  problems 
were  encountered  with  collection  agencies.    The  error  report 
log  is  shown  in  Exhibit  VIII.     Two  types  of  reports  are 
generated : 

The  first  report  lists,  by  hospital,  the  records  found 
to  be  in  error.    Each  is  identified  by  date  of  discharge, 
system  control  number,  and  medical  record  number,  with 
an  English  language  error  message  and  the  associated 
field (s) . 

The  second  is  a  summary  report,  by  hospital,  of  the 
total  number  of  records  received,  the  number  in  error, 
and  a  summary  error  analysis  by  error  description.  (The 
total  volume  reported  is  to  be  checked  against  the 
volume  reported  as  being  submitted  by  the  abstract 
system  and  recorded  on  the  Summary  Control  Log 
described  above.) 

These  reports  were  extremely  helpful  in  identifying  error 
patterns  by  hospital  and  collection  agency  during  the 
initial  months  of  system  operation.    A  supplemental  dump 
report,  which  lists  by  discharge  all  abstract  data  by  field, 
also  assisted  in  identifying  data  recordation,  formatting, 
and  edit  problems  at  the  various  collection  systems. 
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EXHIBIT  G-8 


CO 

z 
o 

H 
Eh 
U 

g 

o 
u 


Q 

w 

Eh 
Eh 

M 

CQ 
D 
CO 


Q 
W 
> 
M 

w 

cj 


as 

Eh 

• 

Z 

o 

Q 

4J 

W 

§ 

0) 

UL 

w 

>1 

Q 

o 

W 

05 

as 

u 

s 

c 

CO 

u 

to 

w 

IS 

J2 

Q 

Eh 

CO  O 

H  W 

t-S  a 
D 

05  Q 

O  W 

«  as 

05  CJ 
W  CO 


o 

Eh 

H 
W  5 

04  Q 

< 
Eh 


O 
O 

B 

O 

8 

0i 

06 
W 


r 
r 
S 

to 

Eh 
CO 
X 
CO 

Eh 
U 

2 

Eh 
CO 
PS 

< 


c 

• 

• 

CQ 

o 

0 

c 

■u 

■P 

o 

0) 

s 

w 

iH 

CM 

CM 

Eh 

rH 

CO 

hS 

a 

CO 

Si 

Eh 

Eh 

Eh 

Eh 

H 

H 

Eh 

H 

M 

04 

04 

04 

CO 

CO 

CO 

CO 

O 

O 

O 

o 

33 

as 

Eh 

as 

as 

CO 

PS 

< 

The  error  reports  are  also  used  for  data  correction  purposes. 
Once  the  error  has  been  identified,  the  correct  entry  is 
indicated  next  to  the  related  record  on  the  error  report 
and  then  resubmitted  to  the  processor  for  correction  of  the 
record.     An  additional  use  for  these  reports  has  been  as  a 
tool  for  retraining  hospital  personnel  in  correct 
abstracting  procedures. 

Monthly  Hospital  Reports 

Certain  reports  from  Dikewood  on  a  hospital  by  hospital 
basis  are  scheduled  to  be  produced  at  monthly  intervals. 
The  log  used  to  monitor  these  reports  is  shown  in  Exhibit 
IX. 

Quarterly,  Semi-Annual  and  Annual  Reports 

For  the  periodic,  PSRO-wide  reports  scheduled  at  quarterly 
and  greater  intervals,  a  log  for  tracking  these  reports 
lists,  by  report: 

Scheduled  date 

Re-scheduled  date 

Receipt  date. 

Abstract  Verification  Quality  Control  Procedures 

Two  approaches  are  being  taken  to  validate  the  quality  of  data 
being  reported  on  the  abstracts: 

Periodic  reabstracting  where  a  sample  of  abstracts 
previously  submitted  is  compared  to  the  medical  record 

Analysis  of  reports  received  from  the  processor  to  identify 
suspicious  coding  patterns;  this  phase  of  the  program  is 
just  being  instituted,  but  it  is  considered  to  be  a  core 
data  quality  activity. 

These  two  approaches  are  described  below. 


Each  quarter,  a  dump  of  50  randomly  selected  records  from 
each  hospital  as  reported  by  the  collection  systems  is  to 
be  received  from  the  processor.     These  serve  as  the  basis 
for  the  reabstracting  sample.    These  are  examined  for  such 
variables  as: 

-        Medical-surgical  mix 
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EXHIP.IT  G-9 
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Case  complexity 


Review  activity  volume,  etc. 

A  final  sample  of  about  15-25  records  per  hospital 
selected  for  review,  the  size  of  the  sample  depending  on 
hospital  size,  number  of  abstractors,  etc.    The  size  of  the 
sample  and  periodicity  of  reabstract ing  is  also  adjusted 
for  other  variables: 

Newly  delegated  hospitals  -  increase  sample  size  and 
frequency 

-  Substandard  past  performance  -  increase  sample  size 
and  frequency 

-  Good  past  performance  -  decrease  frequency  of 
reabstracting  visits. 

As  time  permits,  the  PSRO  plans  to  conduct  interim,  less 
formal  on-site  visits,  especially  when  changes  in  reporting 
requirements  are  required.     Prior  to  the  abstracting  visit, 
the  selected  records,  associated  trailer  abstracts,  and 
coordinator  worksheets  are  retrieved  by  the  hospital. 
During  the  actual  reabstracting,  two  audit  trails  are 
followed : 


The  "P"  elements  from  the  worksheet  to  the  trailer  and 
the  final  abstract 

The  medical  record  data  correspondence  with  the 
reported  UHDDS. 

Any  discrepancy  between  the  recorded  and  reported  data  is 
documented  on  a  case-by-case  basis,  including  the  nature  of 
the  discrepancy  recorded  by  item(s)    involved.    The  review 
of  these  discrepancies  is  not  intended  to  provide  a 
definitive  statistical  analysis,  but  rather  to  identify 
error  patterns  and  determine  their  cause  by  discussions  with 
the  hospital  personnel  involved.    Where  necessary,  technical 
assistance  is  provided  by  hospitals  in  eliminating  prevalent 
problems. 

Where  significant  problems  are  found,  follow-up  interim 
visits  are  planned  to  review  abstracts  being  prepared  at 
the  time.     In  these  cases,  the  PSRO  can  "walk  through"  the 
abstracting  process  with  the  record  and  abstract  in  hand. 
When  a  continuing  problem  is  encountered,  it  will  be  referred 
to  the  Area  XX  Data  Committee  for  possible  hospital 
administration  involvement  in  required  corrective  action. 
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In  some  instances,  hospitals  retain  copies  of  the  abstracts 
submitted  to  their  collection  agency.  These  data  are  then 
compared  against  the  dump  report  to  identify  any  keying  or 
formatting  problems  at  the  collection  agency  level.  This 
process  did,  in  fact,  identify  rather  serious  problems  with 
one  collection  agency. 

A  future  modification  will  apply  to  hospitals  where  more 
than  one  abstractor  is  utilized.     In  these  instances,  each 
abstractor  will  be  required  to  place  some  identifier  on  the 
record.     By  this  mechanism,  the  reabstracting  sample  can  be 
adjusted  to  include  coding  examples  from  all  personnel. 

Reports  Analysis 

In  some  of  the  area  reports  received  by  the  PSRO,  certain 
patterns  have  emerged  which  give  some  indication  that  there 
might  be  coding  problems  not  being  identified  by  the 
reabstracting  process.     Examples  include: 

A  high  incidence  of  low-back  pain  reported  as  a  single 
diagnosis 

A  primary  diagnosis   (single)   of  gastroenteritis  with 
procedures  performed 

A  high  incidence  of  chronic  ischemic  heart  disease  as 
a  primary  diagnosis 

Pneumonia  and  bronchitis  reported  with  procedures 
(chest  x-rays  are  not  codable  procedures) 

Sprains  and  strains  of  the  sacroiliac  with  procedures 
performed 

Closed  reduction  of  fractures  with  procedures  reported. 

Samples  of  these  cases  will  be  verified  for  coding  accuracy. 
Points  to  be  looked  for  include: 

-  Coding  of  admitting  symptoms 
Diagnosis  and  procedures  coding  errors 

-  Underlying  conditions  being  reported  as  primary 
diagnoses. 

This  phase  of  the  program  is  just  beginning.  These  suspected 
problems  may,  in  fact,  be  explainable  in  other  ways;  however, 
it  is  an  issue  felt  to  be  worth  investigation. 
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AREA  XX  DATA  COMMITTEE 


An  important  ingredient  in  the  overall  picture  of  system  operation 
and  data  quality  control  at  Area  XX  is  the  role  of  the  Data  Committee, 
composed  of  three  physicians.    They  view  their  responsibilities  as: 

Effective  operation  of  the  data  system  on  an  on-going  basis 

Data  support  for  other  committees,  such  as  the  Board  of 
Directors,  Quality  Assurance  Committee,  and  Utilization 
Committee. 

Meetings  are  scheduled  on  a  monthly  basis;  however,  if  problems 
arise,  the  chairman  is  on  call  to  provide  support  to  the  PSRO  staff. 
At  each  meeting,  formal  data  system  status  reports  are  reviewed  as 
they  pertain  to: 

Timeliness  of  abstract  submission  to  collection  agencies  by 
all  hospitals  compared  to  performance  over  preceding  months 

Transfer  of  data  from  collection  agencies  to  the  processor 

Processor  performance. 

Each  data  collection  agency  is  reported  and  discussed 
individually.     Resultant  actions  have  included  formal  and  informal 
notification  to  hospitals  regarding  delays  in  abstract  submission, 
coding  problems,  etc.,  and  interaction  with  the  processor.  In  one 
instance,  a  collection  agency  was  notified  that  if  performance  was  not 
improved  within  30  days,  use  of  its  services  to  provide  hospital  data 
would  have  to  be  discontinued.     The  ability  to  invoke  this  action  is 
based  on  a  precaution  included  in  memoranda  of  understanding  with 
delegated  hospitals  which  states  that  if  a  hospital  chooses  to  submit 
data  via  a  collection  agency,  and  should  this  agency  not  conform  to 
the  PSRO  requirements  for  data  timeliness  and  accuracy,  the  PSRO 
reserves  the  right  to  discontinue  the  arrangement  on  30  days  notice 
and  to  require  the  hospital  to  complete  a  PSRO  abstract. 

The  key  to  the  effectiveness  of  the  Data  Committee's  role  is  their 
willingness  to  formally  act  on  behalf  of  the  PSRO  when  staff  efforts 
have  not  resulted  in  the  desired  corrective  action  by  hospitals  and/or 
the  data  collection  agencies  involved  in  the  system. 
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APPENDIX  H 

CASE  STUDY 
SOUTHCENTRAL  PENNSYLVANIA  PSRO 
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CASE  STUDY 


PSRO:         SOUTHCENTRAL  PENNSYLVANIA  PSRQ 

The  following  case  study  describes  the  data  system  and  the  data  quality 
control  program  developed  by  the  Southcentral  Pennsylvania  PSRO  to 
control  the  collection  and  processing  of  the  PHDDS  and  optional  data 
elements  in  a  system  in  which  the  data  is  collected  entirely  through 
existing  data  collection  mechanisms. 

We  are  indebted  to  the  Southcentral  Pennsylvania  PSRO  for  its  support 
and  cooperation  in  this  data  quality  control  endeavor,  and  especially  to 
Harold  Diehl ,  Executive  Director;  Steve  Keys,  Manager,  Data  Systems; 
Yvonne  Harbeson,  Manager  of  Concurrent  Review;  Holly  Hanson,  Data  Clerk; 
and  Sandra  Abnett,  Special  Projects  Coordinator. 

In  addition,  we  are  indebted  to  Arnold  Spellman,  Director,  Project 
Development  and  Roland  Loup,  Manager,  Quality  Control  Department  of  the 
Commission  on  Professional  and  Hospital  Activities  (CPHA);  Thomas  S.  Powell, 
Executive  Director,  Hospital  Utilization  Project  (HUP);  and  his  staff, 
Mr.  William  H.  Schofield;  Ms.  Rosalyn  P.  Robitoille,  R.N.;  Mr.  Mark  Dietz,R.R.A. 
Ms.  Meryl  Hartman,A.R.T. ;  and  Ms.  Carolyn  C.  Petro  for  their  time  and 
assistance  in  preparing  the  case  study. 

PSRO  PERSONNEL  INVOLVED  IN  DATA  ACTIVITIES: 

Harold  Diehl,  Executive  Director 
Stephen  Keys,  Manager,  Data  Systems 
Holly  Hanson,  Data  Clerk 

Yvonne  Harbeson,  Director,  Hospital  Relations  and 

Utilization  Review  Activities 
Sandra  Abnet,  Special  Projects  Coordinator 

ENVIRONMENT 

The  Southcentral  Pennsylvania  PSRO  encompasses  an  eleven-county  area  in 
the  southcentral  part  of  the  state.    There  are  twenty-nine  (29)  acute 
care  hospitals  in  the  area,  all  of  which  have  been  granted  full  delegation 
status.    Of  these  twenty-nine  (29)  hospitals,  six  (6)  participate  in  the 
Professional  Activity  Study  (PAS)  system  of  the  Commission  on  Professional 
and  Hospital  Activities  (CPHA)  and  twenty-three  (23)  participate  in  the 
Hospital  utilization  Project  (HUP).    Since  all  hospitals  in  the  PSRO 
area  participate  in  either  the  PAS  or  HUP  systems,  the  PSRO  is  utilizing 
these  existing  systems  as  the  collection  mechanism  in  its  data  system. 

The  PSRO  is  a  member  of  the  Penn-Del  (Pennsylvania-Delaware)  Data  Consortium. 
This  Consortium  is  comprised  of  seven  (7)  PSROs  from  the  two  (2)  states  that 
share  a  common  framework  for  data  activities.    As  a  result  of  the  Request 
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for  Proposal  (RFP)  issued  by  the  Consortium  members,  the  Hospital  Utilization 
Project  (HUP)  was  selected  as  the  data  processor. 

DATA  SYSTEM 

The  PSRO  data  system  begins  in  the  admissions  office  of  each  delegated 
hospital.    Here,  a  daily  listing  of  Federal  patient  admissions  is  prepared 
for  the  Review  Coordinator. 

The  Review  Coordinator  maintains  individual  patient  worksheets  which 
document  the  concurrent  review  decisions  made  during  the  patient's 
hospital  stay.    Upon  discharge,  the  Review  Coordinator  must  summarize 
the  individual  concurrent  review  information  so  that  the  data  can  be 
transmitted  to  the  medical  records  department  and  properly  entered  on 
the  PAS  or  HUP  abstract.    The  transmission  of  this  information  is 
accomplished  by  several  methods,  depending  on  the  individual  hospital. 
These  methods  include  the  PSRO  developed  "daily  discharge  report," 
(Exhibits  H-l  and  H-2),  unique  hospital  worksheet  or  data  transmittal 
form.    In  several  hospitals,  the  Review  Coordinator  completes  the  con- 
current review  sections  on  the  individual  patient  abstracts. 

The  daily  discharge  report  provides  a  summarization  of  concurrent  review 
information  by  individual  patient  in  the  same  format  as  the  corresponding 
concurrent  review  sections  on  the  HUP  and  PAS  abstracts.    This  report, 
which  is  completed  by  the  Review  Coordinator  on  a  daily  basis,  is  then 
transmitted  to  the  medical  records  department  which,  in  turn,  copies  the 
information  onto  the  patient's  abstract.    Other  hospitals  have  simply 
elected  to  reproduce  the  concurrent  review  section  of  the  abstract  onto 
the  coordinator's  worksheet.    In  all  delegated  hospitals,  the  medical 
records  department  completes  the  UHDDS  elements.    The  final  step  in  the 
concurrent  review  process  is  the  completion  of  the  PSRO  "medical  certification 
form"  (Exhibit  H-3)  by  the  coordinator.    This  certification  form,  which 
summarizes  the  patient's  hospital  stay,  indicates  the  presence  and  scope 
of  PSRO  review  to  the  fiscal  intermediaries. 

Upon  completion,  the  abstracts  are  screened  and  batched  by  medical  records 
department  personnel.    HUP  and  PAS  provide  guidelines  on  batching  procedures 
and  sizes.    At  this  time,  a  control  number  is  assigned  to  each  abstract 
and  a  log  is  kept  as  to  these  numbers.    A  batch  sheet  accompanies  the 
abstracts  to  the  data  service  and  identifies  the  number  of  abstracts  in 
the  batch,  the  date  of  batching  and  the  sequential  numbers  included  in 
the  batch.    Upon  receipt  of  a  batch,  both  HUP  and  PAS  also  log  various 
information  pertaining  to  the  submission.    An  end-of-month  transmittal 
is  enclosed  with  the  last  batch  of  abstracts  for  a  month  so  that  the 
data  abstracting  service  knows  the  month  is  complete.    PSRO  guidelines 
require  that  abstracts  be  completed  and  submitted  within  thirty  (30) 
days  after  discharge. 

The  data  abstracting  services,  which  are  under  contract  to  the  PSRO  as 
data  collectors,  enter  the  logged  abstracts  into  the  computer,  after  the 
material  has  been  scanned  or  verified.    Edits,  as  specified  by  the 
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SOUTHCENTRAL  PENNSYLVANIA  PSRO 
Medical  Arts  Building,  Suite  201 
890  Poplar  Church  Road 
Camp  Hiil,  PA  17011 

MEDICAL  CERTIFICATION  FORM 

Delegated  Hospital   /Area  IX  PSRO 

1.    Patient:   2.    I.D.  #  


3.    Insurance  Class:   4.    Adm.:   D/C: 

5.    PSRO  Termination  Issued:   6.    Order  for  D/C  Effective:   

7.  Comments.  


8.    I  certify  that  this  hospital  stay  has  had  PSRO  review  which  resulted  in  the  following  decisions: 


From  Thru  #  Cert.  Days  Used  LOC  #  Non-Cert.  Days  Note 

From  Thru  #  Cert.  Days  Used  LOC  #  Non-Cert.  Days  .  Note 

From  Thru  #  Cert.  Days  Used  LOC  #  Non-Cert.  Days  Note 

From  Thru  #Cert.  Days  Used  LOC  #  Non-Cert.  Days  Note 

From  Thru  #  Cert.  Days  Used  LOC  #  Non-Cert.  Days  Note 


9.    Total  Number  of  Certified  Days  Used:   10.    Total  Number  Non-Cert.  Days 

11.    Please  check  if  interim  certification:  □ 


Authorized  Representative,  U.R.  Dept. 


Date 


contract,  are  performed  and  errors  are  referred  back  to  the  hospitals 
for  correction.    Once  all  abstracts  for  a  time  period  are  submitted  and 
corrected,  the  collector  prepares  a  computer  tape  which  is  then  forwarded 
to  the  PSRO's  data  processor,  HUP.    HUP  merges  the  data,  performs  the 
necessary  edits  and  prepares  the  PSRO  output  reports  and  the  HSQB  data 
base  tape. 

A  detailed  flowchart  of  the  data  system,  from  the  time  the  patient 
enters  the  hospital  through  the  generation  of  output  reports,  has  been 
developed  and  is  included  on  the  following  page.    (Exhibit  H-4). 

DATA  QUALITY  CONTROL  SYSTEM 

The  Southcentral  Pennsylvania  PSRO  is  keenly  aware  of  the  need  for 
having  an  effective  data  quality  control  program  in  place  to  ensure  the 
accuracy  and  reliability  of  the  data  which  are  used  to  generate  the 
reports  and  profiles  upon  which  the  PSRO  bases  its  management  decisions 
Incorrect  identification,  ordering  or  coding  of  diagnoses  or  procedures 
will  cause  records  to  be  placed  in  wrong  categories.    Missing  records 
can  result  in  erroneous  and  misleading  computations  of  length  of  stay 
rates.    In  order  to  prevent  these  errors  that  reduce  the  reliability  of 
the  data  base,  the  PSRO  is  in  the  process  of  implementing  a  data  quality 
control  program,  which  addresses  the  two  major  aspects  of  data  quality 
control:    data  quality  assurance  procedures  and  data  quality  monitoring 
or  audit  procedures. 

The  data  quality  assurance  procedures  are  those  that  are  built  into  the 
ongoing,  daily  functional  activities  that  comprise  the  PSRO's  data 
system. 

The  monitoring  or  audit  procedures  are  those  that  are  performed  retro- 
spectively on  a  periodic  basis  to  identify  problems,  assess  hospital 
compliance  with  specified  data  quality  procedures  and  standards  and 
verify  the  implementation  of  previously  suggested  corrective  actions. 

Ongoing  Data  Quality  Assurance  Procedures 

The  ongoing  procedures  that  the  PSRO  is  building  into  its  data  system 
are  designed  to  prevent  errors  from  occurring  at  crucial  points  in  the 
system.    These  are: 

Identifying  and  reviewing  all  Federal  patients  subject  to 
PSRO  review, 

Completing  an  abstract  for  each  Federal  patient, 

Accurately  abstracting  and  coding  diagnostic  and  procedural 
data  from  the  medical  record, 

Forwarding  the  correct  number  of  abstracts  to  the  data  collectors 
(CPHA  and  HUP)  from  the  hospital  in  a  timely  manner, 
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Certifying  bills  for  each  Federal  patient  subject  to  review. 

Receiving  all  required  reports  and  deliverables  from  the  processor, 
HUP. 

Identifying  and  Reviewing  All  Federal  Patients 

Federal  patients  are  identified  by  the  admissions  office  when  the  patient 
first  enters  the  hospital.    This  identification  is  handled  in  the  same 
basic  way  in  all  delegated  hospitals.    Upon  admission,  pertinent  informa- 
tion is  obtained  from  every  patient  on  a  hospital's  unique  admission 
sheet.    Once  a  patient  is  identified  as  having  a  Federal  program  (Medicare, 
Medicaid,  Title  V)  as  an  expected  principal  source  of  payment,  a  copy  of 
the  patient's  admission  sheet  is  forwarded  to  the  utilization  review 
department.    This  practice,  combined  with  the  UR  coordinator's  review  of 
the  daily  admission  and  discharge  listings,  assures  that  all  Federal 
patients  are  identified  and  reviewed  on  admission.    In  those  cases  where 
the  admissions  office  feels  the  patient  may  become  a  Federal  beneficiary, 
or  the  patient  has  applied  for  coverage  under  a  Federal  program  (pending 
eligibles),  the  admission  sheets  are  also  forwarded  to  the  UR  department. 
These  patients  will  be  reviewed  concurrently  by  the  UR  coordinator.  If, 
in  some  way,  a  patient  gets  through  the  system  without  being  reviewed, 
still  another  check  on  the  system  exists.    The  PSRO  medical  certification 
form  must  be  completed  by  the  UR  coordinator  on  all  Federal  patients. 
This  form  accompanies  the  bill  to  the  fiscal  intermediary  and  is  their 
indicator  that  the  patient  has  had  PSRO  review.    If  the  certification 
form  is  missing,  the  bill  will  be  referred  back  to  the  hospital.    If  in 
fact  the  patient  was  not  reviewed,  the  UR  coordinator  must  perform  a 
retrospective  review  and  complete  the  medical  certification  form  accordingly. 

Completing  an  Abstract  for  each  Federal  Patient  Discharged 

As  part  of  a  delegated  hospital's  responsibilities  with  respect  to  data, 
a  discharge  abstract  containing  the  PHDDS  and  other  optional  data,  as 
specified  by  the  PSRO,  must  be  completed  on  each  Federal  patient  discharged 
from  the  hospital.    The  PSRO  is  working  with  the  hospitals  to  establish 
procedures  to  assure  compliance  with  this  requirement.    One  such  item 
will  be  the  documentation  of  data-flow  procedures  used  within  the  hospital 
and  the  duties  and  functions  of  each  department  involved  with  the  reporting 
of  PSRO  data.    Admission  screening  and  data  batching  procedures  will 
also  be  documented,  so  that  the  PSRO  can  identify  potential  problem 
areas  and  offer  suggestions.    The  PSRO  will  handle  the  initial  instructions 
in  this  area  and  consult  the  abstracting  services  when  appropriate. 

Accurately  Abstracting  and  Coding  Discharge  Data 

The  Review  Coordinator  is  responsible  for  summarizing  the  review  data 
on  the  Daily  Discharge  Reports.    The  reports  are  then  transmitted  to  the 
medical  records  department  where  medical  records  personnel  transfer  the 
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review  data  (in  a  few  instances  the  Review  Coordinator  transfers  the 
information)  onto  the  discharge  abstract  and  completes  the  UHDDS  portion. 

In  order  to  assure  the  accuracy  of  the  process,  the  following  standards 
and  ongoing  procedures  have  been  (or  are  being)  instituted: 

Education  and  Training 

The  Review  Coordinators  in  the  hospitals  receive  considerable 
training  both  initially  and  on  an  on-going  basis  through  periodic 
PSRO  continuing  education  workshops. 

The  abstracting  organizations  (CPHA  and  HUP)  provide  periodic 
training  and  feedback,  and  continuing  education  relating  to  abstracting 
and  coding  problems  at  the  hospital  for  the  personnel  who  abstract 
the  diagnoses  and  procedures  from  the  medical  records. 

Manuals  and  Instructions 

The  PSRO  is  responsible  for  assisting  the  hospitals  in  procedure 
development,  as  it  relates  to  PSRO  data,  and  for  keeping  the  hospitals 
informed  of  any  definition  or  procedure  changes.    By  means  of  the 
PSRO  administrative  bulletins,  the  hospitals  are  kept  updated  on 
changes  in  procedures,  norms,  criteria,  etc.    The  UR  coordinators 
are  required  to  keep  these  bulletins  in  a  notebook,  which  is  routinely 
screened  by  PSRO  personnel . 

Coding  Standards 

Coding  standards  that  are  being  instituted  are: 
coding  from  a  completed  medical  record, 

encouraging  physicians  to  indicate  the  principal  and  secondary 
diagnoses,  according  to  the  UHDDS  definition,  clearly  in  the 
medical  record  or  on  the  discharge  summary, 

stimulating  physician  interest  in  accurate  coding  by  involving 
them  in  setting  standards  (i.e.,  diagnoses  that  can  never  be 
the  principal  diagnosis), 

criteria  for  ordering  diagnoses  and  procedures, 
instructions  for  handling  questionable  cases. 
Screening 

When  a  hospital  is  granted  PSRO  delegated  review  status,  copies  of 
all  Federal  patient  discharges  are  forwarded  to  the  PSRO  for  a 
minimum  of  a  three-month  period  immediately  following  the  designation 
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of  delegated  status.    These  abstracts  are  screened  by  PSRO  data 
personnel  for  completeness,  reasonableness  and  recurrent  errors  or 
problems.    Manual  concurrent  review  statistics  are  also  gathered 
from  the  data.    After  the  PSRO  is  comfortable  with  the  type  of  data 
being  submitted,  the  hospitals  no  longer  have  to  submit  abstract 
copies. 

Internally,  the  hospitals  are  asked  to  do  the  same  type  of  screening 
before  the  abstracts  are  batched  and  submitted.    HUP  hospitals  use 
a  glassine  overlay  developed  for  this  purpose.    Many  errors  can  be 
identified  and  eliminated  by  this  screening,  and  recurrent  problems 
can  be  focused  in  on  and  corrective  action  taken. 

Editing 

An  ongoing  quality  assurance  procedure  of  the  PSRO's  data  collection 
and  data  processing  system  is  the  system  of  edits.    The  edit  routines 
are  designed  to  identify  errors  in  logic,  invalid  codes  and  fields 
with  missing  data.    Error  lists  are  compiled  and  sent  back  to  the 
hospital  for  correction. 

Forwarding  the  Correct  Number  of  Abstracts  and  Data  Tapes  in  a  Timely  Fashion 

In  order  to  assure  that  abstracts  are  not  lost  during  transit  and  are 
forwarded  in  a  timely  manner  from  the  hospital  to  the  data  collector,  a 
formal  batch  control  system  is  used.    The  procedures  are  established  by 
the  data  abstracting  service  that  the  hospital  participates  in  (HUP  or  PAS). 

Basically,  at  the  time  of  batching,  the  abstracts  are  assigned  a  "control" 
or  "page"  number.    A  batch  contains  sequentially  numbered  abstracts  in  a 
size  determined  by  HUP  or  PAS.    A  batch  cover  sheet  is  prepared  which 
indicates  hospital  identification,  batch  number,  batch  size,  abstract 
sequence,  and  month  and  year  of  discharge. 

Once  batches  are  received,  the  data  collectors  log  in  the  batch,  enter 
the  data  into  the  computer,  perform  the  necessary  edits  and  prepare 
quarterly  data  tapes  for  the  PSRO  data  processor. 

The  PSRO  monitors  the  transfer  of  data  tapes  between  the  data  collector 
and  processor,  and  between  the  processor  and  HSQB.    This  is  done  by 
means  of  a  standard  transmittal  which  provides  specific  inormation  about 
the  data  tapes  being  transferred.    Included  in  this  transmittal  are 
number  of  records  by  hospital  by  month,  data  records  withheld  due  to 
error  and  date  of  tape  generation.    It  should  be  noted  that  since  HUP  is 
both  a  data  collector  and  processor  for  the  PSRO,  such  a  separate  distinction 
and  transfer  is  artificial. 
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In  addition  to  receiving  status  reports  on  the  receipt  of  abstracts  from 
individual  hospitals,  the  PSRO  also  receives  error  recap  reports  on  the 
types  of  errors  being  committed  routinely  by  the  individual  hospitals. 
The  PSRO  uses  this  information  to  monitor  the  timeliness  and  quality  of 
data  being  submitted  and  to  resolve,  when  possible,  delays  in  the  submission 
and  correction  cycle. 

Certifying  Hospitalization  for  All  Federal  Patients 

The  PSRO  is  careful  to  assure  that  hospitalization  is  properly  certified 
and  that  a  copy  of  the  PSRO  medical  certification  form  accompanies  the 
bills  for  every  Federal  patient  discharged  from  a  delegated  hospital. 
As  described  in  the  PSRO's  Review  Coordinator  procedures,  only  UR  coordinators 
or  other  personnel  authorized  by  the  hospital  may  complete  the  medical 
certification  form.    The  fiscal  intermediaries,  in  turn,  check  the 
signatures  on  these  forms  against  a  list  of  authorized  signatures. 
Bills  will  not  be  processed  unless  a  properly  completed  form  is  present. 

A  copy  of  the  completed  form  is  filed  in  the  hospital  for  review  by  the 
PSRO  survey  team.    This  form  is  then  compared  against  the  UR  coordinator 
worksheets  for  accuracy. 

Receiving  All  Required  Reports  and  Deliverables 

In  order  to  monitor  the  timeliness,  quality  and  receipt  of  the  output 
reports  and  other  deliverables  obtained  from  the  data  collectors  and 
processor,  the  PSRO  logs  these  individual  performance  factors.  Items 
logged  include  actual  date  of  submission,  date  of  receipt  and  problems 
encountered,  such  as  rejection  of  HSQB  data  tapes  due  to  an  unacceptable 
error  rate. 

To  assure  that  all  abstracts  sent  to  the  data  collector  and  then  to  the 
processor  are  included  on  the  output  reports,  the  PSRO  compares  totals 
on  the  reports  with  those  in  the  transmittals  sent  to  the  PSRO  on  a 
quarterly  basis.    As  described  earlier,  these  transmittals  indicate 
record  counts  by  hospital  for  the  time  periods  on  the  data  tape.  The 
PSRO  also  compares  these  totals  against  the  total  medical  certification 
forms  submitted  for  a  given  time  period. 

Data  Quality  Audit  Procedures 

The  data  quality  audit  procedures  conducted  by  Southcentral  Pennsylvania 
PSRO  are  directed  at  detecting  problems,  determining  the  source  of  the 
problems,  and  instituting  a  course  for  remedial  action  and  followup  to 
assess  improvement. 

Success  in  carying  out  the  data  audit  procedures  and  instituting  corrective 
actions  has  been  highly  dependent  on  gaining  the  hospital's  cooperation 
and  interest.    The  PSRO,  aware  that  it  is  in  a  key  position  to  affect 
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and  improve  the  quality  of  hospital  discharge  data,  has  worked  towards 
establishing  an  effective  working  relationship  with  the  hospitals.  The 
undertone  of  this  relationship  is  a  cooperative  but  firmly  directive  one 
aimed  at  efforts  to  improve  and  assure  the  quality  of  the  discharge  data 
col lected. 

The  data  quality  audit  procedures,  which  the  PSRO  is  instituting  cooperatively 
with  the  hospitals,  are  used  to  verify  the  following  crucial  aspects  of 
data  quality  control: 

All  Federal  patients  subject  to  PSRO  review  are  identified  and  an 
abstract  prepared. 

The  discharge  data  are  accurately  abstracted  and  coded. 

A  medical  certification  form  is  completed  for  each  Federal  patient 
discharge. 

The  collector  and  processor  are  processing  the  data  according 
to  specified  systems  procedures  and  technical  standards,  i.e., 
required  edits. 

Various  forms  and  reports,  including  several  which  are  available  from 
the  two  data  collection  organizations,  are  used  to  assist  the  PSRO  in 
the  audit  process  which  is  described  in  more  detail  below. 

Verifying  all  Federal  Patients  are  Identified  and  Abstracts  are  Prepared 

In  order  to  verify  that  all  Federal  patients  are  identified,  the  review 
coordinator  checks,  on  a  weekly  basis,  the  worksheets  that  she  maintains 
against  the  daily  admissions  and  discharge  lists  from  the  admitting 
office,  un  a  quarterly  basis,  the  PSRO  is  planning  to  compare  the  listings 
available  for  a  small  additional  charge  from  the  data  collectors  of 
Federal  patient  abstracts  against  the  cumulative  statistics  from  the 
review  coordinator's  daily  log.    A  comparison  of  these  figures  gives  the 
PSRO  an  adequate  approximation  with  which  to  verify  that  the  Federal 
patients  are  being  identified  and  abstracts  are  being  completed  and 
forwarded  to  the  data  collectors. 

The  Discharge  Data  are  Accurately  Abstracted  and  Coded 

Three  audit  procedures  are  to  be  used  to  verify  the  accuracy  of  the  data 
abstracted  from  the  medical  record  and  the  coding.    These  are  (1)  re- 
abstracting,  (2)  use  of  the  error  listings  from  the  data  collectors,  and 
(3)  screening  the  output  reports  for  logic  and  consistency. 

1 .  Re-abstracting 

Re-abstracting  is  carried  out  as  part  of  the  PSRO's  delegated 
hospital  monitoring  activity.    Both  review  activities  and  data 
quality  are  audited  during  quarterly  visits. 
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Being  problem-oriented,  the  following  questions  are  addressed 
during  the  monitoring  process: 

.    Are  the  manuals  containing  instructions  for  forms  completion, 
data  flow,  medical  record  abstracting,  coding,  and  staff 
responsibility  up-to-date,  readily  available,  and  being  used? 

.    Are  procedures,  such  as  required  manual  screening  of  abstracts, 
being  adhered  to? 

.    Are  the  hospitals  completing  the  abstracts  using  the  same  definitions 
for  the  data  items,  and  are  the  definitions  for  the  PHDDS  items, 
the  same  as  those  specified  by  HSQB  for  Federal  reporting  purposes? 

.    Have  the  coders  been  properly  trained,  and  do  they  understand 
the  rules  and  procedures  for  coding,  i.e.,  definitions,  coding 
and  procedures,  and  proper  use  of  the  tabular  list  and  alphabetic 
index  for  accurate  and  complete  coding. 

.  Are  there  patterns  of  errors  which  can  be  traced  to  a  specific 
individual,  i.e.,  a  coder,  a  procedural  deficiency,  a  hospital 
department,  inadequacies  in  medical  record  documentation,  etc.? 

.    What  is  the  best  course  to  correct  the  deficiencies  uncovered? 

.    Have  the  deficiencies  previously  identified  been  corrected? 

Re-abstracting  Procedures 

The  re-abstracting  process  includes  preparation,  on-site  review, 
analysis  of  findings,  and  followup  action.    These  are  discussed  in 
more  detail  below. 

Preparation 

Prior  to  the  visit,  the  PSRO  sets  up  sampling  criteria  to  be  used 
for  selecting  which  medical  records  will  be  used.    Depending  on  the 
size  of  the  hospital  and  past  performance  in  meeting  data  accuracy 
standards,  the  PSRO  re-abstracts  between  ten  and  thirty  medical 
records  from  each  hospital  during  the  monitoring  visit.    The  following 
are  some  of  the  indicators  used  for  selecting  the  records: 

.    Error  feedback  reports  from  the  data  collectors  which  list 
the  errors  identified  during  the  edit  process, 

.    Review  of  routine  statistical  reports  produced  by  the  pro- 
cessor to  determine  areas  of  possible  error,  i.e.,  an 
unusually  high  length  of  stay  rate  or  patient  count  for  a 
particular  diagnosis  or  procedure, 

.    Use  of  post-payment  monitoring  reports  where  discrepancies  in 
certification  have  been  indicated. 
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After  selecting  the  basis  for  sampling,  the  PSRO  notifies  the 
hospital  (usually  the  Director  of  the  Medical  Records  Department) 
of  the  upcoming  visit  and  requests  the  medical  records,  review 
coordinator  worksheets,  logs,  copies  of  medical  certification  forms, 
copies  of  the  abstracts,  and  any  other  relevant  supporting  documentation 
be  pulled  and  made  available  to  the  PSRO  monitoring  team.    In  the 
case  of  hospitals  using  HUP  for  data  collection,  a  copy  of  the 
abstract  remains  in  the  hospital  and  is  available  for  the  PSRO 
monitoring  staff.    The  CPHA  abstracting  system  does  not  leave  a 
copy  of  the  abstract  in  the  hospital.    There  is,  however,  a  listing 
produced  by  CPHA  called  the  "C"-Form  (Exhibit  H-5)  which  is  a 
print-out  of  all  the  critical  information  recorded  on  each  individual 
abstract  record.    Although  it  represents  data  that  has  been  edited 
and  corrected,  the  PSRO  is  able  to  use  portions  of  the  form  in  lieu 
of  the  actual,  original  abstracts  to  conduct  the  reabstracting 
study. 

On-site  Review 

Without  referring  to  the  original  abstract  or  abstract  print-out  on 
the  "C"-Form,  a  re-abstractor  with  medical  records  background 
identifies  all  diagnoses  and  procedures  by  examining  the  entire 
medical  record.    These  are  ordered  and  coded.    The  results  are  then 
verified  against  the  information  on  the  original  abstract.  All 
discrepancies  in  selection,  ordering  or  coding  are  noted  on  the  re- 
abstracting  form,  (Exhibit  H-6).    In  addition,  the  review  data  on 
the  utilization  review  worksheet  is  validated  against  the  data  on 
the  abstract,  as  well  as  on  the  medical  certification  form. 

After  the  monitoring  visit  and  reabstracting  process  are  completed, 
the  PSRO  staff  meets  with  the  review  coordinators  and  appropriate 
medical  records  staff  to  discuss  the  preliminary  findings  and 
possible  reasons  for  the  discrepancies. 

Analysis  of  Findings 

A  complete  analysis  of  the  findings  is  prepared  at  the  PSRO  office 
following  the  monitoring  visit.    This  report  summarizes  the  findings 
of  the  reabstracting  study,  including  the  specific  discrepancies, 
the  probable  reasons  for  the  discrepancies  (i.e.,  judgmental  errors, 
coding  from  inadequate  discharge  summaries,  misuse  of  the  tabular 
indices,  etc.),  and  the  recommendations  for  corrective  actions.  The 
PSRO  forwards  a  copy  of  the  report  analyzing  the  findings  to  each 
respective  hospital . 

A  discrepancy  summary  form  (Exhibit  H-7)  will  be  maintained  for 
each  hospital.    It  will  provide  the  PSRO  with  information  on  each 
hospital's  progress  in  data  quality  assurance  over  time. 
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EXHIBIT  H-6 


Hospital  

SOUTHCENTRAL  PENNSYLVANIA 

Date  

Patient  Medical  Record  No.  PSR0 
  RE-ABSTRACTING  FORM 


Da  t  a    T  t  pm 

Re-abstracting 

UU    aU5  11  (11  L 

G  1  1  U 

Reasons  for 

(l) 

Coding 

vp  —  A  Kef"     r  f~ 

a  riroo  7 

Discrepancies 

(2) 

(3) 

(4) 

Admi ssion 

// 

Clerical 

Date 

//  Completeness 

// 

Procedural 

Di scharge 

// 

Clerical 

Date 

// 

Completeness 

// 

Procedural 

Principal 

Narrative 

// 

Ordering  procedural - 

Diagnosis 

Abstract  Service 

Code 

//  Ordering  procedural  - 

Secondary 

II  \J  j  yJ  I  l*Q  1 

Diagnoses 

1 . 

Narrative 

// 

Ordering  Judgment 

// 

Ordering  other 

Code 

2. 

Narrative 

rnrlinn  nmrpHiiK'P 
l. u u  iiiy        UCCUUI  c 

/  / 

P nHi  nn    i  nHnmoni' 

Code 

/  / 

PnH  inn  flfhoi' 
LUU  i  riy    U  LlICi 

3. 

Narrative 

Code 

4. 

Narrative 

Code 

Principal 

Narrative 

// 

Ordering  procedural- 

Procedure 

Abstract  Service 

Code 

// 

Ordering  procedural - 

Secondary 

hospital 

Procedures 

1 . 

Narrative 

// 

Ordering  judgment 

// 

Ordering  other 

Code 

2. 

Narrative 

// 

Coding  procedure 

// 

Coding  judgment 

Code 

// 

Coding  other 

3. 

Narrative 

Code 

Any  other 

data  items 

that  the  PSRO 

wants  to  audit, 

or  perhaps 

all 

items. 
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Followup  Action 


The  followup  action  depends  on  the  problem  areas  uncovered  in  the 
re-abstracting  study.    It  may  include  any  or  all  of  the  following: 

Requiring  hospitals  to  update  and  standarize  manuals  and 
procedures  (i.e.  requiring  the  use  of  a  uniform  PSRO-specif ied 
definition  for  an  item.) 

Increased  training  and  technical  assistance  through  PSRO 
sponsored  workshops  and  use  of  available  training  and 
technical  assistance  resources  from  the  collection  organiza- 
tions, CPHA  and  HUP. 

In  instances  where  errors  are  traced  back  to  individual 
coders,  intensive  education  or  retraining  of  that  individual. 

Ecouraging  physicians  to  use  more  care  in  completing  the 
medical  records  and  involving  them  in  analyzing  the  output 
reports  from  the  system,  i.e.,  the  profiles,  thereby  motivating 
them  to  improve  the  quality  of  the  discharge  data  abstracted 
from  the  medical  record. 

Informal  periodic  on-site  visits  to  monitor  the  hospitals 
implementation  of  corrective  actions. 

In  addition  to  the  reabstracting  efforts  the  PSRO  also  has  access, 
with  the  hospital's  permission,  to  the  results  of  coding  audits 
performed  periodically  by  HUP. 

2.  Error  Listings 

In  addition  to  reabstracting,  the  PSRO  obtains  error  recap  reports 
on  a  quarterly  basis  which  identify  recurring  errors  by  individual 
hospital.    Although  no  actual  case  numbers  or  identifiers  other 
than  the  hospital  identification  are  provided,  the  information  is 
enough  to  suggest  corrective  actions. 

3.  Reviewing  Output  Reports 

The  output  reports  obtained  from  the  processor  are  reviewed  for 
logic  and  consistency.    Often  error  patterns  can  be  identified  by 
comparing  those  reports  with  specified  criteria  and  parameters. 
For  example,  the  usual  number  of  Title  19  discharges  from  a  particular 
hospital  is  known  to  the  PSRO.    When  the  number  on  the  output 
report  differs  significantly  from  the  expected  number,  the  PSRO 
uses  that  as  an  indicator  of  possible  error  to  be  investigated 
further. 
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Abstracts  are  Collected  and  Processed  in  a  Timely  Manner 

A  third  aspect  of  data  quality  which  the  PSRO's  audit  procedures 
address  is  timeliness.    The  PSRO  requires  the  hospital  to  complete 
and  forward  the  abstracts  on  each  Federal  discharge  to  the  data 
collector  within  30  days  of  discharge.    The  magnetic  tapes  containing 
the  edited  PHDDS  must  be  forwarded  by  the  collectors  to  the  processor 
within  a  sufficient  amount  of  time  to  assure  that  the  PHDDS  magnetic 
tape  can  be  processed  and  submitted  to  HSQB  within  the  required  60 
day  time  limit.    In  order  to  monitor  timeliness,  the  PSRO  receives 
a  monthly  status  report  from  the  data  collectors  which  shows  by 
hospital  the  status  of  input,  the  status  of  previously  indentified 
errors  awaiting  correction,  and  the  last  month  processed  (i.e.,  the 
last  entirely  error-free  month).    When  problems  in  timely  submission 
become  evident,  the  PSRO  contacts  the  hospitals  in  order  to  further 
investigate  the  exact  causes  for  delays  and  assists  in  recommending 
and  implementing  corrective  actions. 

Medical  Certification  Form  is  Completed  for  Each  Federal  Discharge 

Timeliness  and  accuracy  of  bill  certification  are  monitored  during 
the  quarterly  monitoring  visits.    Procedures  and  supporting  documenta- 
tion are  checked.    Copies  of  claims  sent  to  the  payment  agencies 
are  randomly  reviewed  to  verify  PSRO  review  coordinator  certification. 
The  numbers  of  claims  submitted  by  the  hospitals  are  also  verified 
against  the  Review  Coordinator's  daily  discharge  logs.  Descrepancies 
and  problem  areas  noted  are  discussed  with  the  hospital  and  corrective 
action  initiated  as  necessary. 

The  Data  are  being  Processed  According  to  Specified  Systems  Procedures 

A  final  aspect  of  data  quality  which  the  PSRO  audits  involves  the 
reliability  of  the  actual  data  collection  and  processing  systems, 
including  the  accuracy  of  inputting  and  processing  the  data  and 
adherence  to  technical  standards,  such  as  required  edits.  There 
are  three  components  to  verify  the  quality  of  the  systems  procedures 
of  the  data  collector  and  processor:    the  HSQB  and  other  PSRO 
specified  edits,  the  initial  systems  test,  and  periodic  system's 
quality  test.    These  are  described  in  more  detail  below. 

The  HSQB  and  Other  PSRO  Specified  Edits 

The  HSQB  prescribed  logic  edits  are  a  valuable  aid  to  the  PSRO  for 
identifying  problems  or  insufficient  or  inadequate  edit  routines  in 
the  collector's  or  processor's  systems.    Data  tapes  submitted  to 
HSQB  which  do  not  pass  the  edits  specified  in  Transmittals  28,  50 
and  subsequent  updates  are  a  clear  indicator  of  problems  in  the 
system  which  the  PSRO  investigates  and  resolves  as  quickly  as 
possible.    Examination  of  error  listings  and  reports  from  the 
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collectors  and  processors  are  also  used  to  identify  any  processing 
or  edit  difficulties  that  need  to  be  addressed.    For  example,  an 
incorrect  edit  may  be  identified  which  is  responsible  for  the 
system  rejecting  correct  records. 

Initial  Systems  Test 

The  PSRO  is  planning  on  using  the  HSQB  Test  Package,  which  is  a  set 
of  dummy  abstracts  containing  known  errors,  to  test  the  integrity 
of  the  system.    By  entering  and  processing  these  abstracts  through 
the  entire  system,  and  analyzing  the  error  reports  generated,  the 
PSRO  will  be  able  to  assess  whether  the  system  is  working  according 
to  specifications  and  if  the  specified  edits  are  in  place  and  are 
adequately  screening  the  coding  and  logic  errors  in  the  dummy 
abstracts. 

Periodic  Systems  Quality  Testing 

In  addition  to  the  initial  systems  test,  the  PSRO  plans  to  perform 
periodic  systems  tests  to  check  the  ongoing  quality  of  the  collector's 
and  processor's  systems.    These  tests  are  similar  to  the  initial 
systems  test  but  are  performed  on  a  smaller  scale.    They  will  be 
very  useful  in  detecting  problems  and  ensuring  that  all  requested 
modifications  to  the  system  have  been  implemented. 

Summary 

Although  Southcentral  PSRO  obtains  its  data  through  existing  data 
collection  systems,  the  PSRO  is  fully  aware  of  its  responsibility 
to  assure  the  quality  of  the  discharge  data  collected  and  processed. 

To  meet  this  responsibility,  the  PSRO,  with  the  cooperation  of  the 
hospitals  and  the  data  collector  and  data  processor,  has  begun  to 
institute  ongoing  procedures  that  are  part  of  the  routine  data 
system  activities  to  assure  the  quality  of  the  data  and  audit 
procedures  which  are  performed  retrospectively  to  verify  the  reliability 
of  the  data. 

Recognizing  that  without  high  quality  data  the  decisions  that 
affect  the  management,  organization  and  professional  activities  of 
the  program  will  be  inadequate  and  incomplete,  the  PSRO  will  continue 
enforcing  and  refining  its  data  quality  program. 
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